On 1/18/2022 10:16 AM, Brian Gregory wrote:
> On 12/01/2022 11:28, R.Wieser wrote:
>> Hello all,
>>
>> I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
>> copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
>> that the whole thing slows down to a crawl (about 5 GByte consisting of
>> 50.000 files in over eight hours).  I've also tried a another, different USB
>> stick and see the same thing happen.
>
> If you can try with XCOPY.
> XCOPY is good at optimizing the writes to the directory.
> I think it opens and creates all the files first, then copies the contents one by one, and finally closes all the files.
>

If the stick is wiped, and a partition created via Windows 7,
that ensures proper cluster alignment for flash memory storage.
Doesn't matter what file system you use, if the partitions
start on 1MB boundaries, that should help a lot with aligning
the clusters (which are power-of-two), with the flash hardware
addressing (which is power-of-two favoring).

If you cleaned the stick and prepared it on WinXP, then bits of
the geometry are divisible by 63, and that is not a power of two
number, and that doubles the amount of work for writes (assuming
lots of small files, not just one big file).

I've made this kind of error before with hard drives - prepped
them with WinXP, then notice that "something is wrong" on Windows 7.
And eventually, by accident, I might discover my alignment error.
That preparing on Windows XP was a mistake. This is an issue with
512e drives (512 virtual, 4K physical) and not an issue with
512n drives (512 virtual, 512 physical).

I have one drive on this machine right now, WD Gold 4TB, and it is
one of the largest 512n drives you can get. And that means it
doesn't have alignment issues internally (no edge cases, no
read-modify-write). It was purchased in case I needed the drive for
Windows XP, but my Windows XP machine is dead, so now I can use
the 512n drive(s) I have for other things.

Paul

On Tue, 18 Jan 2022 at 13:04:09, Paul <nospam@needed.invalid> wrote (my
responses usually follow points raised):
[]
>If you cleaned the stick and prepared it on WinXP, then bits of
>the geometry are divisible by 63, and that is not a power of two
>number, and that doubles the amount of work for writes (assuming
>lots of small files, not just one big file).
[]
How did the 63 size come about? It seems a very odd choice in computing.
--
J. P. Gilliver. UMRA: 1960/<1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf

Mike Jackson |\ _,,,---,,_
and Squeak /,`.-'`' -. ;-;;,_ Shame there's no snooze button
[1998] |,4- ) )-,_..;\ ( `'- on a cat who wants breakfast
zzz '---''(_/--' `-'\_)

On 1/18/2022 1:31 PM, J. P. Gilliver (John) wrote:
> On Tue, 18 Jan 2022 at 13:04:09, Paul <nospam@needed.invalid> wrote (my responses usually follow points raised):
> []
>> If you cleaned the stick and prepared it on WinXP, then bits of
>> the geometry are divisible by 63, and that is not a power of two
>> number, and that doubles the amount of work for writes (assuming
>> lots of small files, not just one big file).
> []
> How did the 63 size come about? It seems a very odd choice in computing.

It was a choice in CHS.

https://en.wikipedia.org/wiki/Cylinder-head-sector

63 sectors/track

Windows 7 uses unconventional-looking values in the partition table.
You can see these with ptedit32.exe . A utility you should have
acquired back when it was easy to get. The FTP server it was on (for free),
closed some time ago. And since archive.org does not archive FTP sites,
the info is easily lost when files are hosted that way.

Paul

On 1/18/2022 11:42 AM, R.Wieser wrote:
> Brian,
>
>> I think the flash used often has much larger pages than would make sense
>> as an allocation block size.
>
> There is nothing much I can say to that, as its as unspecific as it comes
> ...
>
> Regards,
> Rudy Wieser

8KB is a good value for a 32GB flash stick.

If just means using a custom value, rather than accepting
the default value.

You can take a look through the Micron catalog for
values on some TLC, but then, you won't know which
of the chips is used for USB flash sticks. Presumably
some of the flash is less well tested and suited
to making USB flash sticks. As Micky would say "what
do you expect for $1.50". Someone has to cater to that
market, of "cheap as chips".

I don't think they really throw anything away.
There's always someone vending defective flash, somewhere.
With the bazaar selling model, anything is sustainable.

The TSOP or quad flatpack style chips, might be used
in long-ish USB flash sticks. Wheres the fine pitch BGA
could be used for the stubby USB flash sticks. I only have
one stubby one here, and it sucks. That's the one where I
couldn't find a datasheet for the chip.

Paul

On 1/18/2022 1:51 PM, Paul wrote:
> On 1/18/2022 11:42 AM, R.Wieser wrote:
>> Brian,
>>
>>> I think the flash used often has much larger pages than would make sense
>>> as an allocation block size.
>>
>> There is nothing much I can say to that, as its as unspecific as it comes
>> ...
>>
>> Regards,
>> Rudy Wieser
>
> 8KB is a good value for a 32GB flash stick.
>

Wikipedia says the page size is 4KB to 16KB.

https://en.wikipedia.org/wiki/Flash_memory

The pages can't be too large, because that
would affect the performance of the chip.

Paul

Paul,

> Wikipedia says the page size is 4KB to 16KB.

Which, in short, means three choices : 4K, 8K or 16K , as those numbers need
to be powers of two.

Though that page also mentions something about an "erase page", which size
can be multiple millions of bytes. And somehow that reads as it doesn't
matter if your allocation size matches the page size (unless it matches the
"erasure page" size) ...

Regards,
Rudy Wieser

On 1/19/2022 5:38 AM, R.Wieser wrote:
> Paul,
>
>> Wikipedia says the page size is 4KB to 16KB.
>
> Which, in short, means three choices : 4K, 8K or 16K , as those numbers need
> to be powers of two.
>
> Though that page also mentions something about an "erase page", which size
> can be multiple millions of bytes. And somehow that reads as it doesn't
> matter if your allocation size matches the page size (unless it matches the
> "erasure page" size) ...
>
> Regards,
> Rudy Wieser

A 4K selection (default NTFS) would be a poor choice on an 8K
device, but as long as it has the usual flavor of wear leveling,
the pages can be recycled as needed. Maybe it costs you two writes,
to do 4K clusters on 8K pages, the second write being 8KB of data.
The second half of a page should be write-able, even if it wasn't
used on the first attempt. But if the stick did that, the storage
would be hard to manage.

I would say, having a larger cluster is likely a better idea.

And you can use the RidgeCrop formatter, if you want to customize
the FAT32 on a 128GB USB stick (as Microsoft doesn't do FAT32 above
a certain size).

In the case of NTFS, you can easily select 64KB clusters if you want, but if
you're doing 50000 files, be aware of the "wasted space" problem. I've
used 64K clusters on a backup partition before. For no particular
reason except to try it.

One other thing, regarding the heat issue. It seems some of the
flash chips are 3.3V ones, the USB stick is running off a 5V supply,
and there could be a dropping regulator of some sort to power the flash.
That might be where some of the heat comes from.

Paul

On 19/01/2022 10:38, R.Wieser wrote:
> Paul,
>
>> Wikipedia says the page size is 4KB to 16KB.
>
> Which, in short, means three choices : 4K, 8K or 16K , as those numbers need
> to be powers of two.
>
> Though that page also mentions something about an "erase page", which size
> can be multiple millions of bytes. And somehow that reads as it doesn't
> matter if your allocation size matches the page size (unless it matches the
> "erasure page" size) ...

Why would any size except the "erase page" size be at all relevant to
what we're discussing?

--
Brian Gregory (in England).

Brian,

> Why would any size except the "erase page" size be at all relevant to what
> we're discussing?

Good question. Why don't you start with telling us why it doesn't. ? :-)

Regards,
Rudy Wieser

On 21/01/2022 10:05, R.Wieser wrote:
> Brian,
>
>> Why would any size except the "erase page" size be at all relevant to what
>> we're discussing?
>
> Good question. Why don't you start with telling us why it doesn't. ? :-)

Any other size mismatch can be easily worked around without having to
wait for the Flash chips to do anything.

--
Brian Gregory (in England).

Brian,

>>> Why would any size except the "erase page" size be at all relevant to
>>> what
>>> we're discussing?
>>
>> Good question. Why don't you start with telling us why it doesn't. ? :-)
>
> Any other size mismatch can be easily worked around without having to wait
> for the Flash chips to do anything.

Than you have a problem. I do not know of any computer, Windows or
otherwise, which has a cluster size of a few megabytes. IOW, you will
/always/ have a mismatch in regard to the erase-page size.

Also, *what* other size that can be mismatched ?

And by the way : try to imagine what needs to be done when someone writes a
block matching some other smaller block size into a erase-block sized page
which causes the need to have bits changed from a Zero to a One.

Regards,
Rudy Wieser

On 26/01/2022 15:11, R.Wieser wrote:
> Brian,
>
>>>> Why would any size except the "erase page" size be at all relevant to
>>>> what
>>>> we're discussing?
>>>
>>> Good question. Why don't you start with telling us why it doesn't. ? :-)
>>
>> Any other size mismatch can be easily worked around without having to wait
>> for the Flash chips to do anything.
>
> Than you have a problem. I do not know of any computer, Windows or
> otherwise, which has a cluster size of a few megabytes. IOW, you will
> /always/ have a mismatch in regard to the erase-page size.
>
> Also, *what* other size that can be mismatched ?

If your cluster size is smaller than the minimum block size you can read
or write then you'll need to work around that sometimes but I don't
think you need to erase and re-write anything extra compared to if the
read/write size matches the allocation block size.

> And by the way : try to imagine what needs to be done when someone writes a
> block matching some other smaller block size into a erase-block sized page
> which causes the need to have bits changed from a Zero to a One.

But it does happen less if you make the allocation blocks bigger. Bigger
allocation block size means your erase size blocks get "fragmented" less.

--
Brian Gregory (in England).

Brian,

> If your cluster size is smaller than the minimum block size you can read
> or write then you'll need to work around that

That also needs explanation I'm afraid. What problem (do you think)
(c|w)ould occur and what would the work-around be ?

> but I don't think you need to erase and re-write anything extra compared
> to if the read/write size matches the allocation block size.

Really ? Lets assume that :

1) I write (a cluster) to the latter part of the first allocation block and
the first part of the second allocation block

2) both allocation blocks already contain different data than whats written
to them

3) both allocation blocks are in the same erasure page.

What would the flash chip do?

Now imagine that the cluster size exactly matches the allocation block (and
is aligned with it).

> But it does happen less if you make the allocation blocks bigger. Bigger
> allocation block size means your erase size blocks get "fragmented" less.

While its true its also a red herring. Its not something you or I have
control over. Also, why do you think the producer of those flash chips did
not think of that ? I mean, just make the allocation block the same size as
the erasure page size and presto! no more fragmentation ... :-p

Regards,
Rudy Wieser

R.Wieser <address@not.available> wrote:
[...]

> Than you have a problem. I do not know of any computer, Windows or
> otherwise, which has a cluster size of a few megabytes. IOW, you will
> /always/ have a mismatch in regard to the erase-page size.

Not that it matters one bit for your problem/discussion, but exFAT on
Windows (at least on 8.1) can have allocation unit sizes of upto 32768
kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
16, 32MB).

[...]

Frank,

> but exFAT on Windows (at least on 8.1) can have allocation unit sizes
> of upto 32768 kilobytes, i.e. 32 megabytes.

That doesn't even sound odd to me when looking at the ammount of memory a
current computer can have. IOW, they can probably handle it. Would not
want to try to use that FS on my old C64 though .... :-)

> "a few megabytes" starts at 2MB (then 4, 8, 16, 32MB).

I have to remember that. Many of my files are quite a bit smaller than
that, which would fill up such an FS a bit faster than you would expect
(lots of slack space).

Thanks for the info.

Regards,
Rudy Wieser

"Frank Slootweg" <this@ddress.is.invalid> wrote

|
| Not that it matters one bit for your problem/discussion, but exFAT on
| Windows (at least on 8.1) can have allocation unit sizes of upto 32768
| kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
| 16, 32MB).

Great. So you can fit up to 31 100 KB files on a 1 TB disk. :)

Mayayana <mayayana@invalid.nospam> wrote:
> "Frank Slootweg" <this@ddress.is.invalid> wrote
>
> |
> | Not that it matters one bit for your problem/discussion, but exFAT on
> | Windows (at least on 8.1) can have allocation unit sizes of upto 32768
> | kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
> | 16, 32MB).
>
> Great. So you can fit up to 31 100 KB files on a 1 TB disk. :)

Gimme some of what you're smoking! :-)

MB, not GB.

On 1/27/2022 1:22 PM, Mayayana wrote:
> "Frank Slootweg" <this@ddress.is.invalid> wrote
>
> |
> | Not that it matters one bit for your problem/discussion, but exFAT on
> | Windows (at least on 8.1) can have allocation unit sizes of upto 32768
> | kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
> | 16, 32MB).
>
> Great. So you can fit up to 31 100 KB files on a 1 TB disk. :)
>

It's a capability.

It doesn't mean that it is practical, to select the ExFAT high valued ones.

NTFS on Windows 10 has also been given some larger options.
64KB clusters should work just about anywhere. But the format
menu has some >64KB options only on Windows 10. And I won't be
selecting those, any time soon... unless the choice is backward
compatible all the way back to Win2K. It's no good having partitions
that only mount in Windows 10, and no where else. For one thing,
you want the option of using the CHKDSK from the other OSes,
on occasion.

You can use outlandish choices like that, if your USB stick holds
only MRIMG files from Macrium backups. That's the scenario where
it makes sense.

Paul

"Frank Slootweg" <this@ddress.is.invalid> wrote

| > Great. So you can fit up to 31 100 KB files on a 1 TB disk. :)
| | Gimme some of what you're smoking! :-)
| | MB, not GB.

Woops. Sorry, you're right. So I could save 31,000
small text files. But I couldn't fit the 42K files I have
on my 3 GB backup drive on that 1 TB drive.

On 27/01/2022 01:28, R.Wieser wrote:
> Brian,
>
>> If your cluster size is smaller than the minimum block size you can read
>> or write then you'll need to work around that
>
> That also needs explanation I'm afraid. What problem (do you think)
> (c|w)ould occur and what would the work-around be ?
>
>> but I don't think you need to erase and re-write anything extra compared
>> to if the read/write size matches the allocation block size.
>
> Really ? Lets assume that :
>
> 1) I write (a cluster) to the latter part of the first allocation block and
> the first part of the second allocation block

What are you meaning by allocation block?

I was using the name "allocation block" to mean what also gets called
cluster - the filing system allocation block size.

I guess you mean the block size in the flash.

Firstly I'm assuming the sizes are binary multiples, so no cluster
crosses a boundary between flash blocks.

Secondly I'm assuming that the storage system is managing storage space
and doing wear levelling and will try and arrange things so that when
possible (hopefully most of the time in many cases) block writes are
directed to areas of flash that are in the erased state (you can have a
flash block that is divided up into clusters and leave unused clusters
in the erased state.

--
Brian Gregory (in England).

Brian,

> What are you meaning by allocation block?

:-) What did *you* mean with it ?

> I was using the name "allocation block" to mean what also gets called
> cluster - the filing system allocation block size.

Really ? Than what do you call a *flash* (not FS) read/write block (no, not
the erase-page one) ? 'cause that is what we are talking about, right ?

Also, your message <j4l3s0Fb8r1U1@mid.individual.net> (17-01-22 12:56:47)
seems to contradict the above ...

And why /did/ you chose to use an ambigue name like "allocation block" when
there is, in regard to a harddisk, a perfectly good (and rather distinctive)
"cluster" definition* available ?

* a rather good definition, as I could as easily assume that an "allocation
block" on a harddisk would be referring to a sector.

> I guess you mean the block size in the flash.

Lol ? Which "block size in the flash" are you here referring to ? The
read/write one ? the erase-page one ? Maybe yet another ?

> Firstly I'm assuming the sizes are binary multiples, so no cluster crosses
> a boundary between flash blocks.

See above. It fully depends on what you are referring to with that "flash
block".

On the off chance that you're referring to the flashes read/write page than
the opposite is true - the read/write page matches or is smaller than the FS
cluster thats projected ontop of it.

> Secondly I'm assuming that the storage system is managing storage space
> and doing wear levelling

Its *hard* to respond to an example, isn't it ? It forces you to actually
/think/ about the truths you hold as evident. Better make some
assumptions to make that kind of effort go away. :-\

But as you seem to have no intention to actually answer me it makes no sense
to me to continue our conversation. So, goodbye.

Regards,
Rudy Wieser

On 28/01/2022 13:27, R.Wieser wrote:
> Brian,
>
>> What are you meaning by allocation block?
>
> :-) What did *you* mean with it ?
>
>> I was using the name "allocation block" to mean what also gets called
>> cluster - the filing system allocation block size.
>
> Really ? Than what do you call a *flash* (not FS) read/write block (no, not
> the erase-page one) ? 'cause that is what we are talking about, right ?
>
> Also, your message <j4l3s0Fb8r1U1@mid.individual.net> (17-01-22 12:56:47)
> seems to contradict the above ...
>
> And why /did/ you chose to use an ambigue name like "allocation block" when
> there is, in regard to a harddisk, a perfectly good (and rather distinctive)
> "cluster" definition* available ?
>
> * a rather good definition, as I could as easily assume that an "allocation
> block" on a harddisk would be referring to a sector.

Windows calls what we used to call clusters "Allocation blocks".

J:\>
J:\>
J:\>
J:\>
J:\>chkdsk
The type of the file system is NTFS.
Volume label is Local Disk 1.

WARNING! F parameter not specified.
Running CHKDSK in read-only mode.

CHKDSK is verifying files (stage 1 of 3)...
127232 file records processed.
File verification completed.
13407 large file records processed.
0 bad file records processed.
0 EA records processed.
25 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
139456 index entries processed.
Index verification completed.
0 unindexed files scanned.
0 unindexed files recovered.
CHKDSK is verifying security descriptors (stage 3 of 3)...
127232 file SDs/SIDs processed.
Security descriptor verification completed.
6113 data files processed.
CHKDSK is verifying Usn Journal...
37122192 USN bytes processed.
Usn Journal verification completed.
Windows has checked the file system and found no problems.

1023998975 KB total disk space.
999706604 KB in 63950 files.
36132 KB in 6114 indexes.
0 KB in bad sectors.
263007 KB in use by the system.
65536 KB occupied by the log file.
23993232 KB available on disk.

4096 bytes in each allocation unit.
255999743 total allocation units on disk.
5998308 allocation units available on disk.

J:\>
J:\>
J:\>
J:\>

See.

>
>> I guess you mean the block size in the flash.
>
> Lol ? Which "block size in the flash" are you here referring to ? The
> read/write one ? the erase-page one ? Maybe yet another ?

It's up to you to be less ambiguous.

>
>> Firstly I'm assuming the sizes are binary multiples, so no cluster crosses
>> a boundary between flash blocks.
>
> See above. It fully depends on what you are referring to with that "flash
> block".

It should apply to any of them.

>
> On the off chance that you're referring to the flashes read/write page than
> the opposite is true - the read/write page matches or is smaller than the FS
> cluster thats projected ontop of it.

Then I have no idea what point you were trying to make.

If you think that those blocks are smaller then no read modify write is
required.

>
>> Secondly I'm assuming that the storage system is managing storage space
>> and doing wear levelling
>
> Its *hard* to respond to an example, isn't it ? It forces you to actually
> /think/ about the truths you hold as evident. Better make some
> assumptions to make that kind of effort go away. :-\
>
> But as you seem to have no intention to actually answer me it makes no sense
> to me to continue our conversation. So, goodbye.

Stupid moron.

--
Brian Gregory (in England).