Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

"America is a stronger nation for the ACLU's uncompromising effort." -- President John F. Kennedy


devel / comp.compression / ZX0 variant only okay...

SubjectAuthor
* ZX0 variant only okay...Harry Potter
`* Re: ZX0 variant only okay...Harry Potter
 `* Re: ZX0 variant only okay...Harry Potter
  `* Re: ZX0 variant only okay...Harry Potter
   `* Re: ZX0 variant only okay...Harry Potter
    `- Re: ZX0 variant only okay...Harry Potter

1
ZX0 variant only okay...

<1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=424&group=comp.compression#424

  copy link   Newsgroups: comp.compression
X-Received: by 2002:ac8:4588:0:b0:3bf:a99f:474e with SMTP id l8-20020ac84588000000b003bfa99f474emr1846640qtn.0.1678562327336;
Sat, 11 Mar 2023 11:18:47 -0800 (PST)
X-Received: by 2002:a05:6214:1764:b0:56b:fe74:d652 with SMTP id
et4-20020a056214176400b0056bfe74d652mr726831qvb.7.1678562327209; Sat, 11 Mar
2023 11:18:47 -0800 (PST)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Sat, 11 Mar 2023 11:18:47 -0800 (PST)
Injection-Info: google-groups.googlegroups.com; posting-host=74.108.229.93; posting-account=xRocggoAAACFej4w6sQauoZjUP9yroE5
NNTP-Posting-Host: 74.108.229.93
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com>
Subject: ZX0 variant only okay...
From: rose.joseph12@yahoo.com (Harry Potter)
Injection-Date: Sat, 11 Mar 2023 19:18:47 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 1828
 by: Harry Potter - Sat, 11 Mar 2023 19:18 UTC

Hi! I'm working on several compression techniques, and most of them usually do pretty well. Some of them are variants of the ZX0 compression technique for 8-bit systems. One of them is doing only okay, but another one is doing very well on most files. The last one mentioned does significantly better on most files but significantly worse on one. :( I am adding the following to the base technique:

* Adaptive Huffman codes
* My own replacement of the Elias technique for writing lengths and part of LZ77 offsets
* Last16, which shortens repeated LZ77 blocks to an offset to the previous based on # blocks to the copy
* Shorten literals if same as last two literals
* A different way to do BPE

I am asking for ideas to add to these. Thank you for listening.

Re: ZX0 variant only okay...

<2b8fb9cd-407c-4d4c-a22d-ecfe78c39bf4n@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=586&group=comp.compression#586

  copy link   Newsgroups: comp.compression
X-Received: by 2002:ac8:5e53:0:b0:3e3:8bbd:b367 with SMTP id i19-20020ac85e53000000b003e38bbdb367mr4209499qtx.7.1682874116592;
Sun, 30 Apr 2023 10:01:56 -0700 (PDT)
X-Received: by 2002:a05:622a:1885:b0:3ef:1c64:a9ff with SMTP id
v5-20020a05622a188500b003ef1c64a9ffmr4217495qtc.10.1682874116466; Sun, 30 Apr
2023 10:01:56 -0700 (PDT)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Sun, 30 Apr 2023 10:01:56 -0700 (PDT)
In-Reply-To: <1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=2603:7000:153f:bfe0:2db6:7a20:241d:cc6f;
posting-account=xRocggoAAACFej4w6sQauoZjUP9yroE5
NNTP-Posting-Host: 2603:7000:153f:bfe0:2db6:7a20:241d:cc6f
References: <1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <2b8fb9cd-407c-4d4c-a22d-ecfe78c39bf4n@googlegroups.com>
Subject: Re: ZX0 variant only okay...
From: rose.joseph12@yahoo.com (Harry Potter)
Injection-Date: Sun, 30 Apr 2023 17:01:56 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 2035
 by: Harry Potter - Sun, 30 Apr 2023 17:01 UTC

Okay. I added some optimizations to Adaptive Huffman codes that sometimes updates the Huffman code tree when writing an LZ77 or Last16 block or a reference to one of the last two literals (a derivative of my Placement Offset Basic technique--ask for more information). I also have some optimizations to LZ77 that will decrease the size of an LZ77 block if doing so produces better results. Right now, this technique does 2.6% better than Exomizer. My goal is 3% or better by the end of the day. I could apply the rllz technique mentioned in a previous post here. I plan to try it out at a later date but will need a way to identify which bytes are being filled by the rllz blocks. For now, I'm looking for other ways to do better. If you have any suggestions, I'd be happy.

Re: ZX0 variant only okay...

<b4f16d7e-314c-481b-86e4-a6637ce7d88dn@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=587&group=comp.compression#587

  copy link   Newsgroups: comp.compression
X-Received: by 2002:a05:620a:4f:b0:74e:324:d6eb with SMTP id t15-20020a05620a004f00b0074e0324d6ebmr1930775qkt.7.1682875262477;
Sun, 30 Apr 2023 10:21:02 -0700 (PDT)
X-Received: by 2002:a05:622a:1006:b0:3f0:a9a0:45e3 with SMTP id
d6-20020a05622a100600b003f0a9a045e3mr4201393qte.12.1682875262317; Sun, 30 Apr
2023 10:21:02 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!1.us.feeder.erje.net!feeder.erje.net!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Sun, 30 Apr 2023 10:21:01 -0700 (PDT)
In-Reply-To: <2b8fb9cd-407c-4d4c-a22d-ecfe78c39bf4n@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=2603:7000:153f:bfe0:2db6:7a20:241d:cc6f;
posting-account=xRocggoAAACFej4w6sQauoZjUP9yroE5
NNTP-Posting-Host: 2603:7000:153f:bfe0:2db6:7a20:241d:cc6f
References: <1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com> <2b8fb9cd-407c-4d4c-a22d-ecfe78c39bf4n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <b4f16d7e-314c-481b-86e4-a6637ce7d88dn@googlegroups.com>
Subject: Re: ZX0 variant only okay...
From: rose.joseph12@yahoo.com (Harry Potter)
Injection-Date: Sun, 30 Apr 2023 17:21:02 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 1583
 by: Harry Potter - Sun, 30 Apr 2023 17:21 UTC

I'm also not compressing using Adaptive Huffman codes if doing so will result in larger data, am sometimes updating the Huffman tree if repeating one of the last two literals in literal compression and making certain literal compression techniques optional if not being used in a block of data.

Re: ZX0 variant only okay...

<a69cdbbf-4461-4d9d-adb4-fd405654c602n@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=588&group=comp.compression#588

  copy link   Newsgroups: comp.compression
X-Received: by 2002:a05:620a:19a7:b0:74f:b51b:3de4 with SMTP id bm39-20020a05620a19a700b0074fb51b3de4mr189579qkb.5.1683284745518;
Fri, 05 May 2023 04:05:45 -0700 (PDT)
X-Received: by 2002:a05:620a:1a90:b0:74d:f7d0:6a57 with SMTP id
bl16-20020a05620a1a9000b0074df7d06a57mr236168qkb.12.1683284745365; Fri, 05
May 2023 04:05:45 -0700 (PDT)
Path: i2pn2.org!i2pn.org!news.1d4.us!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Fri, 5 May 2023 04:05:44 -0700 (PDT)
In-Reply-To: <b4f16d7e-314c-481b-86e4-a6637ce7d88dn@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=74.108.229.93; posting-account=xRocggoAAACFej4w6sQauoZjUP9yroE5
NNTP-Posting-Host: 74.108.229.93
References: <1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com>
<2b8fb9cd-407c-4d4c-a22d-ecfe78c39bf4n@googlegroups.com> <b4f16d7e-314c-481b-86e4-a6637ce7d88dn@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <a69cdbbf-4461-4d9d-adb4-fd405654c602n@googlegroups.com>
Subject: Re: ZX0 variant only okay...
From: rose.joseph12@yahoo.com (Harry Potter)
Injection-Date: Fri, 05 May 2023 11:05:45 +0000
Content-Type: text/plain; charset="UTF-8"
X-Received-Bytes: 1428
 by: Harry Potter - Fri, 5 May 2023 11:05 UTC

I also have a version that uses MTF instead of Adaptive Huffman. Right now, that version is doing way too well to be true. In case it is true, what can I add to MTF to make it better?

Re: ZX0 variant only okay...

<8069fb9d-60f0-46b4-aa8b-7dcb91002cc1n@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=589&group=comp.compression#589

  copy link   Newsgroups: comp.compression
X-Received: by 2002:a05:622a:447:b0:3ef:3395:de6a with SMTP id o7-20020a05622a044700b003ef3395de6amr401849qtx.6.1683286958495;
Fri, 05 May 2023 04:42:38 -0700 (PDT)
X-Received: by 2002:a05:620a:3715:b0:74e:2e3:8e24 with SMTP id
de21-20020a05620a371500b0074e02e38e24mr240588qkb.6.1683286958366; Fri, 05 May
2023 04:42:38 -0700 (PDT)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Fri, 5 May 2023 04:42:38 -0700 (PDT)
In-Reply-To: <a69cdbbf-4461-4d9d-adb4-fd405654c602n@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=74.108.229.93; posting-account=xRocggoAAACFej4w6sQauoZjUP9yroE5
NNTP-Posting-Host: 74.108.229.93
References: <1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com>
<2b8fb9cd-407c-4d4c-a22d-ecfe78c39bf4n@googlegroups.com> <b4f16d7e-314c-481b-86e4-a6637ce7d88dn@googlegroups.com>
<a69cdbbf-4461-4d9d-adb4-fd405654c602n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <8069fb9d-60f0-46b4-aa8b-7dcb91002cc1n@googlegroups.com>
Subject: Re: ZX0 variant only okay...
From: rose.joseph12@yahoo.com (Harry Potter)
Injection-Date: Fri, 05 May 2023 11:42:38 +0000
Content-Type: text/plain; charset="UTF-8"
X-Received-Bytes: 1458
 by: Harry Potter - Fri, 5 May 2023 11:42 UTC

I was wrong, and I'm sorry. :'( I was exiting a loop at the first byte in a block compressed. I am still doing better but nearly as much better than I thought.

Re: ZX0 variant only okay...

<049eb618-a577-401f-a95c-7257f4f4fdc2n@googlegroups.com>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=658&group=comp.compression#658

  copy link   Newsgroups: comp.compression
X-Received: by 2002:a05:6214:9a9:b0:626:15f7:623a with SMTP id du9-20020a05621409a900b0062615f7623amr1736796qvb.0.1687186648759;
Mon, 19 Jun 2023 07:57:28 -0700 (PDT)
X-Received: by 2002:a05:622a:1891:b0:3fd:e156:77bb with SMTP id
v17-20020a05622a189100b003fde15677bbmr2813527qtc.11.1687186648495; Mon, 19
Jun 2023 07:57:28 -0700 (PDT)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Mon, 19 Jun 2023 07:57:28 -0700 (PDT)
In-Reply-To: <8069fb9d-60f0-46b4-aa8b-7dcb91002cc1n@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=74.108.229.93; posting-account=xRocggoAAACFej4w6sQauoZjUP9yroE5
NNTP-Posting-Host: 74.108.229.93
References: <1760a481-2322-4a88-8785-47ca74ad7371n@googlegroups.com>
<2b8fb9cd-407c-4d4c-a22d-ecfe78c39bf4n@googlegroups.com> <b4f16d7e-314c-481b-86e4-a6637ce7d88dn@googlegroups.com>
<a69cdbbf-4461-4d9d-adb4-fd405654c602n@googlegroups.com> <8069fb9d-60f0-46b4-aa8b-7dcb91002cc1n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <049eb618-a577-401f-a95c-7257f4f4fdc2n@googlegroups.com>
Subject: Re: ZX0 variant only okay...
From: rose.joseph12@yahoo.com (Harry Potter)
Injection-Date: Mon, 19 Jun 2023 14:57:28 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 2126
 by: Harry Potter - Mon, 19 Jun 2023 14:57 UTC

I'm doing much better with this variant now and am almost ready to start optimizing and debugging. But first, I want to buy a few more points with the compression ratio. I have ways to optimize lz77: somebody online told me about hash tables; while processing this information, I thought to use an 8k buffer of bits, where each bit determines whether an associated word was already encountered; somebody else helped me optimize the main loop, and I was testing the current word in the lz77 sliding dictionary before checking for a match. Are there any other ways I can optimize lz77? I am working for 8-bit computers as well as more modern computers, so memory may sometimes be at a premium.

1
server_pubkey.txt

rocksolid light 0.9.8
clearnet tor