Frank,

> Sorry, a case of sloppy reading! I thought you didn't understand
> Herbert's German text.

I'm actually from his west-side next-door neighbour, the Netherlands.

A lot of "nachsynchronisierter schaukasten films" (hope I wrote that right)
in my youth and a year getting at school enabled me to read most of it.

> I will now crawl back under my rock. :-)

Ah, you do not need to go /that/ far... :-p

Regards,
Rudy Wieser

Char,

> I suppose so, but I'd say that's preferable to the alternative, which
> is not knowing

Hmmm... not knowning versus losing everything. I think I would go for not
knowing.

> I suppose each file would have to be loaded in some way
....
> but all of the major container types give you an easy method to
> detect corruption of the container itself

Which happens by reading the whole container and than doing some checking
magic. A difference between lots of small parts versus one big blob.

Though the multiple small parts would need a seperate file to store all the
checksums. Either that, or add them as an alternate data stream to the
files themselves (which ofcourse would need the storage to be NTFS or
comparable in this regard)

> Right, you have to plan ahead. You have to ask yourself, are these
> files something that I care about?

Well, that is why I'm making backups of them, aren't I ? :-)

> If so, how much damage do I want to be prepared to correct?

I'm not prepared to repair anything.

On the other hand, I'm making multiple backups of stuff thats important to
me. If one even /hints/ of failing I make a new copy from it (or from one
of the others) - and compare it against one of the others to verify its
integrity.

> Quickpar creates additional files

Ah yes, ofcourse. For some reason I was thinking of the origional data
being wrapped in the parity/checksom/recovery data. Or that the data was
stored in a before mentioned alternate data stream.

Regards,
Rudy Wieser

On Fri, 14 Jan 2022 23:00:46 +0100, "R.Wieser" <address@not.available>
wrote:

>Char,
>
>> I suppose so, but I'd say that's preferable to the alternative, which
>> is not knowing
>
>Hmmm... not knowning versus losing everything. I think I would go for not
>knowing.

That's a false choice, obviously. :-)

The choice is knowing or not knowing. It's much, much easier to check a
container (and by doing so, check all of its contents in one fell
swoop), than to check individual files.

It sounds like you have a solution that works for you, though, so it's
all good.

<snip>

Char,

> That's a false choice, obviously. :-)
>
> The choice is knowing or not knowing.

I don't think so. But lets not pursue this further, shall we.

> It's much, much easier to check a container (and by doing so,
> check all of its contents in one fell swoop), than to check
> individual files.

:-) Its not you doing the checking, but a program. And just as you can let
your containerizing program do the checking for you so can I start my
program to trawl thru all those files of mine and do the same.

> It sounds like you have a solution that works for you, though, so
> it's all good.

I was thinking the same.

Thanks for bringing your methods up though. It gave me other possibilities
to consider. Which is always good.

Regards,
Rudy Wieser

On Wed, 12 Jan 2022 12:28:35 +0100, "R.Wieser" <address@not.available>
wrote:

>I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
>copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
>that the whole thing slows down to a crawl (about 5 GByte consisting of
>50.000 files in over eight hours). I've also tried a another, different USB
>stick and see the same thing happen.

I use a batch file to do something similar, in order to copy data
files from my desktop computer (XP) to my laptop (Win7), and back
again.

Some of the files are compressed (ARJ or ZIP) and some are not.

Sometimes, especially on the XP machine, the USB connection drops in
the middle of a transfer, and then I need to run CHKDSK on the flash
drive, otherwise it slows down or might lose data.

Perhaps you could try running CHKDSK /f

--
Steve Hayes from Tshwane, South Africa
Web: http://www.khanya.org.za/stevesig.htm
Blog: http://khanya.wordpress.com
E-mail - see web page, or parse: shayes at dunelm full stop org full stop uk

Steve,

> Sometimes, especially on the XP machine, the USB connection drops in
> the middle of a transfer, and then I need to run CHKDSK on the flash
> drive, otherwise it slows down or might lose data.

Thanks for the suggestion. Someone else mentioned that his stick heats up
while writing and causes problems like I described. Maybe yours has the
same problem ?

After I posted my above question and not getting a "Thats a well-known
problem" response I've done a number of tests. As it turns out the problem
is that specific USB stick (see the further posts I made)

As such I decided to effectivily bin it (I can still muck around with it,
but it won't ever be used for storage again)

Regards,
Rudy Wieser

R.Wieser <address@not.available> wrote:
> Frank,
>
> > Sorry, a case of sloppy reading! I thought you didn't understand
> > Herbert's German text.
>
> I'm actually from his west-side next-door neighbour, the Netherlands.

Yes, I know that, like me, you're a Dutchie. In the beginning your
lastname had me thinking you were German, but that was a long time ago.

> A lot of "nachsynchronisierter schaukasten films" (hope I wrote that right)
> in my youth and a year getting at school enabled me to read most of it.

I hate(d) those films. Earth to Germany: John Wayne isn't (wasn't) a
German. And - in news programs - neither is Joe Biden. There's this
thing called 'subtitles', you should try it some time! :-(

> > I will now crawl back under my rock. :-)
>
> Ah, you do not need to go /that/ far... :-p

Well, it's actually quite cosy! :-)

Frank,

> Yes, I know that, like me, you're a Dutchie. In the beginning your
> lastname had me thinking you were German, but that was a long time ago.

Well, I've been told that my great-granddad (or one above that) came from
Germany to try his luck here, and just never left. IOW, do seem to have
some german blood in me.

> I hate(d) those films. Earth to Germany: John Wayne isn't (wasn't) a
> German. And - in news programs - neither is Joe Biden.

nowerdays I have the same dislike towards televised movies. Not much of a
problem with having some dubbed stuff coming thru the news though.

> There's this thing called 'subtitles', you should try it some time! :-(

Gee, what an innovative idea. :-p

But yes, I take that over audio dubbing any time. Than again, I've always
found subtitles handy to support my understanding of what gets spoken.

>> > I will now crawl back under my rock. :-)
>>
>> Ah, you do not need to go /that/ far... :-p
>
> Well, it's actually quite cosy! :-)

Well yeah, but it hurts my back when I have to bow down that deep to look at
you when I want to talk to you. :-)

Regards,
Rudy Wieser

On 12/01/2022 13:52, R.Wieser wrote:
> Paul,
>
> Thanks for the suggestions.
>
> The reason why I copy files is that I can than access them using any kind of
> filebrowser - even a simple one which doesn't understand the concept of ZIP
> folders.
>
> As for the cluster size, the second USB stick was new, and as such I assume
> it was pre-formatted with the optimum block size. Also, it was NTFS (I had
> hoped it would make a difference, but it looks like it doesn't)
>
> And there is a problem with that suggestion : Years ago, when I thought of
> the same (micro-SD connected to a DOS 'puter), I was unable to get that
> optimum from the USB stick itself. No matter which drive interrogation
> function I looked at, none of them returned the size of the FLASH chips
> memory blocks . :-(

As a rule of thumb always format USB flash drives with 32K allocation
blocks. A possible alternative is to remember what size blocks were used
when it was brand new. By default Windoze tends to format USB flash with
blocks that are much too small.

--
Brian Gregory (in England).

Brian,

> As a rule of thumb always format USB flash drives with 32K allocation
> blocks.

Thanks. I've heard that number before, but never thought of it as an
universal one (true for sticks of any make and size). I've always assumed
that the flash-chips block size could/would change with the chips size.

> A possible alternative is to remember what size blocks were used when it
> was brand new.

You mean that I have to store those values for all my different (and
changing) sticks in a document somewhere (or a sticking it on the stick
itself) ? Yeah, I thought about that too. But as I cannot even seem to
be able to uniquely mark an individual stick (so I can look it up in a list)
....

> By default Windoze tends to format USB flash with blocks that are much too
> small.

And there I was, assuming that MS would know more about this that I could
ever hope to do, and therefore use some sensible default. :-(

Thanks for the info.

Regards,
Rudy Wieser

"R.Wieser" <address@not.available> wrote

| > As a rule of thumb always format USB flash drives with 32K allocation
| > blocks.
| | Thanks. I've heard that number before, but never thought of it as an
| universal one (true for sticks of any make and size). I've always
assumed
| that the flash-chips block size could/would change with the chips size.
|

It's not universal. Block size only relates to the limits
of 4-byte integers. I don't know whether the counting is
an unsigned or signed number, but the blocks have to
be big enough that the number of them doesn't end up
more than the limit of a long integer on a 32-bit system.

| > A possible alternative is to remember what size blocks were used when it
| > was brand new.
| | You mean that I have to store those values for all my different (and
| changing) sticks in a document somewhere (or a sticking it on the stick
| itself) ? Yeah, I thought about that too. But as I cannot even seem to
| be able to uniquely mark an individual stick (so I can look it up in a
list)
| ...
| | > By default Windoze tends to format USB flash with blocks that are much
too
| > small.
| | And there I was, assuming that MS would know more about this that I could
| ever hope to do, and therefore use some sensible default. :-(
| | Thanks for the info.
| | Regards,
| Rudy Wieser
| |

On 1/16/2022 8:39 AM, Mayayana wrote:
> "R.Wieser" <address@not.available> wrote
>
> | > As a rule of thumb always format USB flash drives with 32K allocation
> | > blocks.
> |
> | Thanks. I've heard that number before, but never thought of it as an
> | universal one (true for sticks of any make and size). I've always
> assumed
> | that the flash-chips block size could/would change with the chips size.
> |
>
> It's not universal. Block size only relates to the limits
> of 4-byte integers. I don't know whether the counting is
> an unsigned or signed number, but the blocks have to
> be big enough that the number of them doesn't end up
> more than the limit of a long integer on a 32-bit system.

I hoped to have two examples (using dead USB sticks),
but one of them, the datasheet is untraceable.

**********************************************
Lexar S73 32GB (Not exactly fast)

29f128g08cfaaa 16GBx2

Open NAND Flash Interface (ONFI) 2.2-compliant
Multiple-level cell (MLC) technology
Organization
Page size x8: 8640 bytes (8192 + 448 bytes) <=== the 448 bytes could be ECC
Block size: 256 pages (2048K + 112K bytes)
Plane size: 2 planes x 2048 blocks per plane
Device size: 64Gb: 4096 blocks;
128Gb: 8192 blocks; <===
256Gb: 16,384 blocks;
512Gb: 32,786 blocks
Endurance: 5000 PROGRAM/ERASE cycles

Array performance
Read page: 50μs (MAX)
Program page: 1300μs (TYP)
Erase block: 3ms (TYP)

Synchronous I/O performance
Up to synchronous timing mode 5
Clock rate: 10ns (DDR)
Read/write throughput per pin: 200 MT/s
**********************************************

I would think a cluster size of 8K or larger
would be a start, to reduce write amplification.

Note that Windows 7 preparation of the stick, should
align the beginning of the partition to 1MB (0x100000)
boundaries. And by aligning the partition to binary-power-of-two,
that avoids the OS trying to write "the end of one page and
a bit onto the second page".

Don't format the sticks on Windows XP, for fear or
screwing up the alignment.

Then do the experiment again.

The second chip I wanted to research was this,
but I got zero reasonable leads for this search.

***************************************
This is a 128GB miniature USB flash stick, with only
a single Ball Grid Array flash chip on it. Can't find a
datasheet, to get a page size. The marking on the chip
of "NW605" is not a part number, but seems to reference
the MT29F part number below.

Lexar S23-128 plastic barrel, 10MB/sec writes

3XA22 NW605 (BGA)

MT29F1T08CUEABH8-12:A
NAND Flash 1TBIT 83MHZ 152LBGA

This is one of those "really shouldn't be a USB3 product" designs.
The plastic barrel for the USB stick was a disaster, allowing
one of the USB3 high speed pins to snap off during insertion,
and dooming the device to reading at USB2 rates.
***************************************

I have other USB sticks that are still working, but on those
the trick to disassembling them without damaging them is not obvious
to me.

Paul

Mayayana,

> It's not universal. Block size only relates to the limits
> of 4-byte integers. I don't know whether the counting is
> an unsigned or signed number, but the blocks have to
> be big enough that the number of them doesn't end up
> more than the limit of a long integer on a 32-bit system.

Are you sure ? I mean, why would a flash chip do anything with individual
byte addressing over its whole range ?

As far as I know the reading/writing of the flash chip is broken up in two
parts : one command with an address to indicate which block needs to be
read/written, and another command with its own address to read bytes
from/write bytes to the flash-chips internal buffer.

As long as that buffer is smaller than what the OS can handle everything is
hunky-dory. Which might well be why the blocksize is limited to "just" 32
KByte.

Regards,
Rudy Wieser

P.s.
I just checked, and the largest "allocation size" I can choose is a mere
4096 bytes ....

R.Wieser <address@not.available> wrote:
> P.s.
> I just checked, and the largest "allocation size" I can choose is a mere
> 4096 bytes ....

Maybe that's an XP limitation.

FWIW, on my Windows 8.1 system for a 4GB SD-card (i.e. not an USB
memory stick) the choices range from 1024 bytes to 32 kilobytes for
FAT32 and from 512 bytes to 64 kilobytes for NTFS.

What was your stick-size and type of filesystem again? (I lost track
in this long thread.)

Frank,

> What was your stick-size and type of filesystem again? (I lost
> track in this long thread.)

Its an 8 GByte stick currently NTFS formatted.

Regards,
Rudy Wieser

On 2022-01-16, R.Wieser <address@not.available> wrote:

>> By default Windoze tends to format USB flash with blocks that are much too
>> small.
>
> And there I was, assuming that MS would know more about this that I could
> ever hope to do, and therefore use some sensible default. :-(

Sensible default? Microsoft? Thanks - I needed a laugh this morning.

--
/~\ Charlie Gibbs | Microsoft is a dictatorship.
\ / <cgibbs@kltpzyxm.invalid> | Apple is a cult.
X I'm really at ac.dekanfrus | Linux is anarchy.
/ \ if you read it the right way. | Pick your poison.

On 1/16/2022 1:42 PM, Charlie Gibbs wrote:
> On 2022-01-16, R.Wieser <address@not.available> wrote:
>
>>> By default Windoze tends to format USB flash with blocks that are much too
>>> small.
>>
>> And there I was, assuming that MS would know more about this that I could
>> ever hope to do, and therefore use some sensible default. :-(
>
> Sensible default? Microsoft? Thanks - I needed a laugh this morning.
>

For formatting large FAT32 volumes (up to 2.2TB), the Ridgecrop
fat32formatter can do that. This solves the ~32GB limit Microsoft
has on fat32 partitions. I think there might be some cluster
size options as well.

https://web.archive.org/web/20200424145132/http://www.ridgecrop.demon.co.uk/index.htm?fat32format.htm

https://web.archive.org/web/20200410224838if_/http://www.ridgecrop.demon.co.uk/download/fat32format.zip

*******

Windows 10 has some extra-large cluster size options for
NTFS, but those are not backward compatible with NTFS
on earlier OSes, and I can't really recommend using
a 1MB cluster on Win10 and then not being able to
read it on Windows 7. Clusters up to 64KB might be
a bit more compatible across the OSes.

*******

4KB clusters are used on C: , so that Compression and
EFS encryption will work. Windows 10 C: used to support
having 64KB clusters on it. But after the third or
fourth upgrade, it stopped "tolerating" that choice.
I only discovered this by accident, by taking an older
partition and installing Windows 10 in it... without
reformatting first. Oops :-)

Since you are not normally installing Windows 10 C: on
a USB flash drive, this 4KB choice is not an issue.
The Windows To Go thing, might have been the reason
there was some room for non-standard clusters at first.
Once WTG was removed as an option, that would make it
easier for them to insist on 4KB clusters.

https://docs.microsoft.com/en-us/windows/deployment/planning/windows-to-go-overview

Paul

On 16/01/2022 16:48, R.Wieser wrote:
> Frank,
>
>> What was your stick-size and type of filesystem again? (I lost
>> track in this long thread.)
>
> Its an 8 GByte stick currently NTFS formatted.
>
> Regards,
> Rudy Wieser
>
>

If you want it to work fast don't use NTFS. Use FAT or exFAT.

--
Brian Gregory (in England).

On 16/01/2022 08:20, R.Wieser wrote:
> Brian,
>
>> As a rule of thumb always format USB flash drives with 32K allocation
>> blocks.
>
> Thanks. I've heard that number before, but never thought of it as an
> universal one (true for sticks of any make and size). I've always assumed
> that the flash-chips block size could/would change with the chips size.

It probably does change. But we're not needing an exact match to get a
speed benefit.

Having a big block on the drive divided into many small allocation
blocks seems to be what can slow things down.

--
Brian Gregory (in England).

Brian,

> If you want it to work fast don't use NTFS. Use FAT or exFAT.

I only reformatted to NTFS to check if the problem was perhaps files-system
related. It doesn't seem to be.

> It probably does change. But we're not needing an exact match to get a
> speed benefit.

Speed is one thing. Not needing reading-modify-write the same flash-block
is another. And that will happen if the allocation size is not the same
as, or a multiple of, the flash-block size.

> Having a big block on the drive divided into many small allocation blocks
> seems to be what can slow things down.

:-) you or the stick needing to do multiple read-modify-write actions will
/definitily/ slow the stick down more than a simple write action - even if
you just divide the flash block in (an uneven) two.

IOW, I do understand the mechanics behind it. Which is why I'm (still) a
bit surprised you cannot ask the stick for its optimum allocation size ...

Regards,
Rudy Wieser

On 1/15/2022 5:03 AM, R.Wieser wrote:
> Steve,
>
>> Sometimes, especially on the XP machine, the USB connection drops in
>> the middle of a transfer, and then I need to run CHKDSK on the flash
>> drive, otherwise it slows down or might lose data.
>
> Thanks for the suggestion. Someone else mentioned that his stick heats up
> while writing and causes problems like I described. Maybe yours has the
> same problem ?
>
> After I posted my above question and not getting a "Thats a well-known
> problem" response I've done a number of tests. As it turns out the problem
> is that specific USB stick (see the further posts I made)
>
> As such I decided to effectivily bin it (I can still muck around with it,
> but it won't ever be used for storage again)
>
> Regards,
> Rudy Wieser

Might be interesting to open it up and add a heat sink and see what
happens.

Mike,

> Might be interesting to open it up and add a heat sink and see what
> happens.

I thought of that too. But after I stopped the copying, ejected it, left it
alone for an hour (letting it cool down), restarted the copy and than saw it
was directly going back into its one-file-each-six-seconds rithm I didn't
think that that would give me any more information.

Regards,
Rudy Wieser

On 17/01/2022 16:17, R.Wieser wrote:
> Brian,
>
>> If you want it to work fast don't use NTFS. Use FAT or exFAT.
>
> I only reformatted to NTFS to check if the problem was perhaps files-system
> related. It doesn't seem to be.
>
>> It probably does change. But we're not needing an exact match to get a
>> speed benefit.
>
> Speed is one thing. Not needing reading-modify-write the same flash-block
> is another. And that will happen if the allocation size is not the same
> as, or a multiple of, the flash-block size.
>
>> Having a big block on the drive divided into many small allocation blocks
>> seems to be what can slow things down.
>
> :-) you or the stick needing to do multiple read-modify-write actions will
> /definitily/ slow the stick down more than a simple write action - even if
> you just divide the flash block in (an uneven) two.
>
> IOW, I do understand the mechanics behind it. Which is why I'm (still) a
> bit surprised you cannot ask the stick for its optimum allocation size ...
>

I think the flash used often has much larger pages than would make sense
as an allocation block size.

--
Brian Gregory (in England).

On 12/01/2022 11:28, R.Wieser wrote:
> Hello all,
>
> I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
> copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
> that the whole thing slows down to a crawl (about 5 GByte consisting of
> 50.000 files in over eight hours). I've also tried a another, different USB
> stick and see the same thing happen.

If you can try with XCOPY.
XCOPY is good at optimizing the writes to the directory.
I think it opens and creates all the files first, then copies the
contents one by one, and finally closes all the files.

--
Brian Gregory (in England).

Brian,

> I think the flash used often has much larger pages than would make sense
> as an allocation block size.

There is nothing much I can say to that, as its as unspecific as it comes
....

Regards,
Rudy Wieser