Late last week, a script that I have used for several years suddenly
stopped working. Investigation showed that wget was failing to
download some pages. A simplified version, showing the problem, is:

$ cat ic
uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
wget "https://www.marketwatch.com/investing/index/spx"" rel="nofollow" target="_blank">https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500
$ . ./ic
--2024-03-11 08:52:23-- https://www.marketwatch.com/investing/index/spx
Resolving www.marketwatch.com (www.marketwatch.com)... 13.227.37.29, 13.227.37.70, 13.227.37.8, ...
Connecting to www.marketwatch.com (www.marketwatch.com)|13.227.37.29|:443... connected.
HTTP request sent, awaiting response... 401 HTTP Forbidden

Username/Password Authentication Failed.
$

Looking at the error message, one might think that this page/site
requires user login credentials. However, the same URL works just
fine in Firefox, with no login requested or required.

Despite this, I tried telling wget to provide empty username and
password, with no observable change in results.

On a purely cargo-cult basis, I tried some different user agent
strings, with no effect.

I searched on "401 HTTP Forbidden", only to find that there does
not appear to be such an error. There is "401 Unathorized", and
"403 Forbidden", but no such cross-breed.

I looked briefly at the page source (in Firefox), but without a
top-level design document, couldn't make head or tail of it.

Does anybody have any suggestions on how to fix my problem and
again automatically download this, and neighboring, pages?
--
Michael F. Stemper
Indians scattered on dawn's highway bleeding;
Ghosts crowd the young child's fragile eggshell mind.

On 2024-03-11, Michael F. Stemper wrote:
> Late last week, a script that I have used for several years suddenly
> stopped working. Investigation showed that wget was failing to
> download some pages. A simplified version, showing the problem, is:
>
> $ cat ic
> uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
> wget "https://www.marketwatch.com/investing/index/spx"" rel="nofollow" target="_blank">https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500
> $ . ./ic
> --2024-03-11 08:52:23-- https://www.marketwatch.com/investing/index/spx
> Resolving www.marketwatch.com (www.marketwatch.com)... 13.227.37.29, 13.227.37.70, 13.227.37.8, ...
> Connecting to www.marketwatch.com (www.marketwatch.com)|13.227.37.29|:443... connected.
> HTTP request sent, awaiting response... 401 HTTP Forbidden
>
> Username/Password Authentication Failed.
> $
>
> Looking at the error message, one might think that this page/site
> requires user login credentials. However, the same URL works just
> fine in Firefox, with no login requested or required.

Looks like the page *does* have a login button / javascript thing
"somewhere" (at least I can see it when I open the page in lynx here).
I'd imagine either

(1) wget is respecting some robots.txt somewhere OR
(2) wget is following that login link for some reason

The "401" is the error code. The "HTTP Forbidden" is (for lack of a
better word) "custom text" they're supplying. I've done similar where a
HTTP upload process sends back "200 OK, Got it!" as a proof-of-sanity
when scripting things with expect.

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On 11.03.24 15:27, Dan Purgert wrote:
> On 2024-03-11, Michael F. Stemper wrote:
>> Late last week, a script that I have used for several years suddenly
>> stopped working. Investigation showed that wget was failing to
>> download some pages. A simplified version, showing the problem, is:
>>
>> $ cat ic
>> uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
>> wget "https://www.marketwatch.com/investing/index/spx"" rel="nofollow" target="_blank">https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500
>> $ . ./ic
>> --2024-03-11 08:52:23-- https://www.marketwatch.com/investing/index/spx
>> Resolving www.marketwatch.com (www.marketwatch.com)... 13.227.37.29, 13.227.37.70, 13.227.37.8, ...
>> Connecting to www.marketwatch.com (www.marketwatch.com)|13.227.37.29|:443... connected.
>> HTTP request sent, awaiting response... 401 HTTP Forbidden
>>
>> Username/Password Authentication Failed.
>> $
>>
>> Looking at the error message, one might think that this page/site
>> requires user login credentials. However, the same URL works just
>> fine in Firefox, with no login requested or required.
>
> Looks like the page *does* have a login button / javascript thing
> "somewhere" (at least I can see it when I open the page in lynx here).
> I'd imagine either
>
> (1) wget is respecting some robots.txt somewhere OR
> (2) wget is following that login link for some reason
>
> The "401" is the error code. The "HTTP Forbidden" is (for lack of a
> better word) "custom text" they're supplying. I've done similar where a
> HTTP upload process sends back "200 OK, Got it!" as a proof-of-sanity
> when scripting things with expect.
>
Besides that ... is it on purpose that $uas is between single quotes, so
won't get expanded? Double quotes are required because the user agent
string has blanks (and parantheses), but single quotes are definitely
wrong here!

Josef "2cts" Möllers

On 3/11/2024 10:11 AM, Michael F. Stemper wrote:
> Late last week, a script that I have used for several years suddenly
> stopped working. Investigation showed that wget was failing to
> download some pages. A simplified version, showing the problem, is:
>
> $ cat ic
> uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
> wget "https://www.marketwatch.com/investing/index/spx"" rel="nofollow" target="_blank">https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500
> $ . ./ic
> --2024-03-11 08:52:23--  https://www.marketwatch.com/investing/index/spx
> Resolving www.marketwatch.com (www.marketwatch.com)... 13.227.37.29, 13.227.37.70, 13.227.37.8, ...
> Connecting to www.marketwatch.com (www.marketwatch.com)|13.227.37.29|:443... connected.
> HTTP request sent, awaiting response... 401 HTTP Forbidden
>
> Username/Password Authentication Failed.
> $
>
> Looking at the error message, one might think that this page/site
> requires user login credentials. However, the same URL works just
> fine in Firefox, with no login requested or required.
>
> Despite this, I tried telling wget to provide empty username and
> password, with no observable change in results.
>
> On a purely cargo-cult basis, I tried some different user agent
> strings, with no effect.
>
> I searched on "401 HTTP Forbidden", only to find that there does
> not appear to be such an error. There is "401 Unathorized", and
> "403 Forbidden", but no such cross-breed.
>
> I looked briefly at the page source (in Firefox), but without a
> top-level design document, couldn't make head or tail of it.
>
> Does anybody have any suggestions on how to fix my problem and
> again automatically download this, and neighboring, pages?
Almost like there is a mixing up at some point, of https://
versus http:// in the operation. The website denying http:// access.

Maybe at some point, the website used to redirect the http://
attempt to https:// for you, and maybe it's not doing that
any more ?

Or perhaps wget has developed a defect in dining habits
related to that aspect.

Paul

On 11/03/2024 11.46, Paul wrote:
> On 3/11/2024 10:11 AM, Michael F. Stemper wrote:

>> $ cat ic
>> uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
>> wget "https://www.marketwatch.com/investing/index/spx"" rel="nofollow" target="_blank">https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500
>> $ . ./ic
>> --2024-03-11 08:52:23--  https://www.marketwatch.com/investing/index/spx
>> Resolving www.marketwatch.com (www.marketwatch.com)... 13.227.37.29, 13.227.37.70, 13.227.37.8, ...
>> Connecting to www.marketwatch.com (www.marketwatch.com)|13.227.37.29|:443... connected.
>> HTTP request sent, awaiting response... 401 HTTP Forbidden
>>
>> Username/Password Authentication Failed.
>> $

> Almost like there is a mixing up at some point, of https://
> versus http:// in the operation. The website denying http:// access.
>
> Maybe at some point, the website used to redirect the http://
> attempt to https:// for you, and maybe it's not doing that
> any more ?

Sorry, but I don't follow this. The URL that I show above is
https:// not http:// and that's also what the output of wget
shows as the URL.

What is the source for your suspicion that it's really doing
http:// under the covers?

--
Michael F. Stemper
There's no "me" in "team". There's no "us" in "team", either.

On 11/03/2024 11.08, Josef Möllers wrote:
> On 11.03.24 15:27, Dan Purgert wrote:
>> On 2024-03-11, Michael F. Stemper wrote:
>>> Late last week, a script that I have used for several years suddenly
>>> stopped working. Investigation showed that wget was failing to
>>> download some pages. A simplified version, showing the problem, is:
>>>
>>> $ cat ic
>>> uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
>>> wget "https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500

> Besides that ... is it on purpose that $uas is between single quotes, so won't get expanded? Double quotes are required because the user agent string has blanks (and parantheses), but single quotes are definitely wrong here!

Interesting. All this time, I've been sending a User Agent string
of $uas, and it's worked.

If I recall my thinking from six years ago, I had used single quotes
because I used double quotes in the definition of the variable. Now
that I look back, that was pretty obviously wrong, since the *value*
of the variable doesn't have any quotes in it.

However, changing from single to double didn't help. (I'm guessing
that you didn't expect that it would.)

Single versus double quotes always get me tangled up.

--
Michael F. Stemper
There's no "me" in "team". There's no "us" in "team", either.

On 11/03/2024 09.27, Dan Purgert wrote:
> On 2024-03-11, Michael F. Stemper wrote:
>> Late last week, a script that I have used for several years suddenly
>> stopped working. Investigation showed that wget was failing to
>> download some pages. A simplified version, showing the problem, is:
>>
>> $ cat ic
>> uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
>> wget "https://www.marketwatch.com/investing/index/spx"" rel="nofollow" target="_blank">https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500
>> $ . ./ic
>> --2024-03-11 08:52:23-- https://www.marketwatch.com/investing/index/spx
>> Resolving www.marketwatch.com (www.marketwatch.com)... 13.227.37.29, 13.227.37.70, 13.227.37.8, ...
>> Connecting to www.marketwatch.com (www.marketwatch.com)|13.227.37.29|:443... connected.
>> HTTP request sent, awaiting response... 401 HTTP Forbidden
>>
>> Username/Password Authentication Failed.
>> $
>>
>> Looking at the error message, one might think that this page/site
>> requires user login credentials. However, the same URL works just
>> fine in Firefox, with no login requested or required.
>
> Looks like the page *does* have a login button / javascript thing
> "somewhere" (at least I can see it when I open the page in lynx here).

I've never installed lynx. Is it capable of running as a background
process, e.g., via crontab?

> I'd imagine either
>
> (1) wget is respecting some robots.txt somewhere OR
> (2) wget is following that login link for some reason

Any ideas how I could test for, or prevent, either of these?

--
Michael F. Stemper
There's no "me" in "team". There's no "us" in "team", either.

On 2024-03-11, Michael F. Stemper wrote:
> On 11/03/2024 09.27, Dan Purgert wrote:
>> On 2024-03-11, Michael F. Stemper wrote:
>>> Late last week, a script that I have used for several years suddenly
>>> stopped working. Investigation showed that wget was failing to
>>> download some pages. A simplified version, showing the problem, is:
>>>
>>> $ cat ic
>>> uas="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
>>> wget "https://www.marketwatch.com/investing/index/spx"" rel="nofollow" target="_blank">https://www.marketwatch.com/investing/index/spx" -U '$uas' -O SP500
>>> $ . ./ic
>>> --2024-03-11 08:52:23-- https://www.marketwatch.com/investing/index/spx
>>> Resolving www.marketwatch.com (www.marketwatch.com)... 13.227.37.29, 13.227.37.70, 13.227.37.8, ...
>>> Connecting to www.marketwatch.com (www.marketwatch.com)|13.227.37.29|:443... connected.
>>> HTTP request sent, awaiting response... 401 HTTP Forbidden
>>>
>>> Username/Password Authentication Failed.
>>> $
>>>
>>> Looking at the error message, one might think that this page/site
>>> requires user login credentials. However, the same URL works just
>>> fine in Firefox, with no login requested or required.
>>
>> Looks like the page *does* have a login button / javascript thing
>> "somewhere" (at least I can see it when I open the page in lynx here).
>
> I've never installed lynx. Is it capable of running as a background
> process, e.g., via crontab?

Not that I'm aware of, sorry.

>
>> I'd imagine either
>>
>> (1) wget is respecting some robots.txt somewhere OR
>> (2) wget is following that login link for some reason
>
> Any ideas how I could test for, or prevent, either of these?

Potentially adding "-e robots=off" will avoid #1. More verbosity (-v) or
turning on headers (-S?) may help for both as well.

But both of these were a bit of a stab in the dark.

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On 12/03/2024 04.24, Dan Purgert wrote:
> On 2024-03-11, Michael F. Stemper wrote:
>> On 11/03/2024 09.27, Dan Purgert wrote:
>>> On 2024-03-11, Michael F. Stemper wrote:
>>>> Late last week, a script that I have used for several years suddenly

>>>> Looking at the error message, one might think that this page/site
>>>> requires user login credentials. However, the same URL works just
>>>> fine in Firefox, with no login requested or required.
>>>
>>> Looks like the page *does* have a login button / javascript thing
>>> "somewhere" (at least I can see it when I open the page in lynx here).

>>> I'd imagine either
>>>
>>> (1) wget is respecting some robots.txt somewhere OR
>>> (2) wget is following that login link for some reason
>>
>> Any ideas how I could test for, or prevent, either of these?
>
> Potentially adding "-e robots=off" will avoid #1. More verbosity (-v) or
> turning on headers (-S?) may help for both as well.

No joy from robots=off, and wget's man page says that -v is the default.

But, I just tried with curl, and think that I've found a clue. Included
in what it downloaded was:
"Please enable JS and disable any ad blocker"

I'm not sure if it's possible for wget to fake having javascript, but
it seems as if that's the next place to look.
--
Michael F. Stemper
This sentence no verb.