You guys have similar interests to me. Have you found a good way to
organise your electronic files?

I don't mean just programs and documentation but all the files we store
over a lifetime - correspondence, data, downloads, photos, videos,
business dealings, personal records etc.

I'm fairly sure it's best to organise by topic but what should the
hierarchy be?

Any suggestions?

--
James Harris

On 2021-05-28 12:38, James Harris wrote:
> You guys have similar interests to me. Have you found a good way to
> organise your electronic files?
>
> I don't mean just programs and documentation but all the files we store
> over a lifetime - correspondence, data, downloads, photos, videos,
> business dealings, personal records etc.
>
> I'm fairly sure it's best to organise by topic but what should the
> hierarchy be?
>
> Any suggestions?

This is one of the cases where relational paradigm is better than
hierarchical one.

I am not aware if there exist filesystems allowing multiple and
relational views. [File/node links do not count, of course]

I still remember earlier times (:-)), there existed extensions for DEC's
PDP-11 FILESYSTEM-11 where you could annotate files and directories and
then run searches over the annotations.

That would be the closest thing, IMO.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On Fri, 28 May 2021 11:38:12 +0100
James Harris <james.harris.1@gmail.com> wrote:

(server prohibits alt. crosspost, separating reply - c.l.m)

> You guys have similar interests to me. Have you found a good way to
> organise your electronic files?

No, not really. I'm very ad-hoc in this regard, but I do organize as
needed. It consumes time and effort to organize. There is more
stuff to organize than I have time for, just as there is more work
than I have time for. So, things get "filed" everywhere. I also
noticed that my interests shift, so what files I'm actually using
changes too. I find that it's not usually a big problem, if I don't
have access to older files. It is mostly the psychology trauma, i.e.,
psychological "loss".

> I don't mean just programs and documentation but all the files we
> store over a lifetime - correspondence, data, downloads, photos,
> videos, business dealings, personal records etc.

The one thing I've come to adopt over the years is to have everything
of mine underneath one top-level directory. This keeps "my stuff"
separate from operating system files, collections such as music,
videos, or pdf specifications from the internet, etc. I started this
after I had trouble backing up a system, because my files were
everywhere.

Of course, if you recall the file counts reply to another post of yours,
I've not been very good at following that in all circumstances, as I've
got pdfs everywhere ...

> I'm fairly sure it's best to organise by topic but what should the
> hierarchy be?

AIR, from the file counts, our categories of electronic files were
quite similar. Those categories are probably the best starting point
for an organized hierarchy that I've got.

I just name directories whatever suits me, whatever fits. I do the same
with browser folder names for bookmarks too. I find it's easier to
remember where some stuff is stored by certain names than by other
equivalent names.

When I find I'm having problems repeatedly remembering where something
is stored, then I'll rename the difficult to locate folder or directory
to what I keep thinking that I named it, or to what I keep thinking
that I should've named it.

Also, I usually remember acronyms for all types of stuff without ever
having to memorize the acronym. So, using acronyms works well for me.
Abbreviations too. I have trouble remembering personal names, place
names, dates, phone numbers, addresses, etc. Acronyms help with names,
e.g., you are J.H. for James Harris, but so is James Hetfield of
Metallica, as well as many other J.H. named people, etc. If I've met
the person in real life, I usually memorize their face (an image) and
two initials together, which helps me remember their full name, as I'm a
visual thinker. Cryptic names or names not familiar to me don't work
well. With Linux (instead of DOS), I have the option of longer names,
which helps for certain topics.

> [choice of backup storage medium]

Well, I switched from CD-ROM backups to external USB ... Much more
storage space nowadays. No need to compress data. No need to
span or split data or archives across multiple disks. Faster to read
from. Easier to manage files or retrieve them.

I'm not sure about the data permanence or longevity of an HD vs CD-ROM
vs SDD. I know floppies and tapes (audio cassette, VCR, reel-to-reel)
are terrible for long-term storage. Supposedly, USB sticks aren't very
good either. I haven't noticed losses with other media types, but I
also haven't had to do a restore from backups either. I have had many
of one brand of HD fail, but I've been fortunate in that I've been able
to copy off the data so far, without reverting to backups. Although, I
do usually do a backup just prior to an upgrade so I can reinstall.

Basically, I've saved almost all of my data since the start of the
CD-ROM and IDE HD era. The earlier stuff on floppies and MFM HDs is
mostly lost.

So, I think the best option is to just keep backing stuff up to new
high-capacity media every few years to reduce or prevent data loss, and
keep upgrading your primary storage devices too. Then, you have
multiple redundant copies of data of different ages, on different media
types, with different data-loss rates. I wouldn't be too excessive
though. There is no need to keep too many copies around.

--
The SALT deduction is a kickback of taxes to wealthy people in wealthy
states.

On 2021-05-29 14:59, Rod Pemberton wrote:

> I'm not sure about the data permanence or longevity of an HD vs CD-ROM
> vs SDD. I know floppies and tapes (audio cassette, VCR, reel-to-reel)
> are terrible for long-term storage.

I can read 30 years old floppies. Digital bands must be even better,
AFAIK you must rewind them periodically.

> I have had many
> of one brand of HD fail, but I've been fortunate in that I've been able
> to copy off the data so far, without reverting to backups.

Everyone of my Seagate HDDs failed except two very old devices for 90s!

> Although, I
> do usually do a backup just prior to an upgrade so I can reinstall.

I use ZFS on top of RAID-1 pairs, with SMART activated and scrubs
running. That should never fail undetected.

> Basically, I've saved almost all of my data since the start of the
> CD-ROM and IDE HD era. The earlier stuff on floppies and MFM HDs is
> mostly lost.

Same here, though in my case all floppies are readable, even 5" ones!
You must have stored them in a very dry place, I guess.

> So, I think the best option is to just keep backing stuff up to new
> high-capacity media every few years to reduce or prevent data loss, and
> keep upgrading your primary storage devices too. Then, you have
> multiple redundant copies of data of different ages, on different media
> types, with different data-loss rates. I wouldn't be too excessive
> though. There is no need to keep too many copies around.

I agree.

Unfortunately LTO tapes never became usable in terms of bucks per GB. I
would consider LTO if you somehow could get a device for free (5-10K).
LTOs alone are little bit less expensive than HDDs and much more
durable, if you store them properly.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On Sat, 29 May 2021 15:05:55 +0200
"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote:

> On 2021-05-29 14:59, Rod Pemberton wrote:

> > I have had many
> > of one brand of HD fail, but I've been fortunate in that I've been
> > able to copy off the data so far, without reverting to backups.
>
> Everyone of my Seagate HDDs failed except two very old devices for
> 90s!

In my case, it is/was all of my Western Digital IDE HDs, and also MFM
HDs. All of my non-WD IDE drives still work.

--
The SALT deduction is a kickback of taxes to wealthy people in wealthy
states.

On 5/28/21 4:38 AM, James Harris wrote:
> You guys have similar interests to me. Have you found a good way to
> organise your electronic files?

I don't know if it qualifies as /good/ or not, but I have a things that
I do.

> I don't mean just programs and documentation but all the files we store
> over a lifetime - correspondence, data, downloads, photos, videos,
> business dealings, personal records etc.

I can't tell if you're talking about physical files or electronic files.

> I'm fairly sure it's best to organise by topic but what should the
> hierarchy be?

Sort of.

But first what is a topic? For books, is it the title, or the author,
or the publisher, or where you acquired it, or keywords, or???

> Any suggestions?

I have long been a fan of (symbolic) links to directories and / or files
in a hierarchy. I will often have a hierarchy that is based on
manufacturer (particularly of software).

I can, and occasionally do have another hierarchy that is just
application name, mixing multiple vendors.

Sometimes I have another hierarchy that is specific projects and just
links to the things that I need for a given project.

I find that the (symbolic) links work great for electronic files as it
allows the files to appear in more than one hierarchy.

I can even have the different appearances of a file have different names
in different hierarchies.

Physical things are more problematic as I can't have multiple
instantiations of them. -- Well, I can, but we're not going down the
duplication path. -- I tend to have books sorted by broad topic;
Linux, networking, reference (dictionaries, etc.). I tend to group
things within the broad topic; all Linux OS / Distributions, Linux
Programs, Linux tutorials, etc.

I'm in the slow process of constructing a database for some things;
think notes, how-tos, etc, in a green-field creation of a database. I
plan on having the ability to tag things with categories and multiple
views that are different categories sorted differently. (A common
current example is adding labels to email inside of Gmail.) The
database that I'm using supports replication between clients and
multiple servers. Replication means offline support. I can even have
my phone be a client, including offline support. The database also
supports encryption at the database level and at individual field level.
There's also a web interface to a database via the server.

Storing things electronically has a lot of advantages.

I'm planing on using my new custom database as a way to bridge the
physical / electronic divide. I'm thinking seriously about scanning
physical documents at a moderate resolution (which can be easily read,
maybe even reproduced with enough quality to stand up in court, it will
depend on space) so that I can easily do the multiple instantiation that
I was talking about above. I can also include a field that includes the
physical document number and the last location that it was in. So, I'm
able to do the bulk of my work with electronic files while still falling
back to the physical file if / when the need arises.

For example, I can scan a bill from a doctor and instantiate it under
that month's bills, the doctor, the patient (family member),
chronological log, medical condition (eye glasses), etc. I can also
easily share references to the document with my wife. All from the
single electronic copy of the scan of the physical document.

Oh, ya, I'll host it myself in my house, and likely not allow anything
outside the house to access it. -- Well, there will be an off site
replica at a good friends house whom I trust. -- Remember that
replication ~> offline access, and encryption? }:-)

The database & server product also includes / uses PKI primitives to
control who is allowed to access the database and / or documents therein.

--
Grant. . . .
unix || die