On 02 Jul 2024, Daniel said the following...
Da> If I attempt accessing it with lynx, I get this generic error:
Da> "Due to aggressive automated scraping of FederalRegister.gov and
Da> eCFR.gov, programmatic access to these sites is limited to access to our
Da> extensive developer APIs."
Da> I don't get this message on a normal javascript enabled browser. I
Da> surmise it views queries from non-javascript browsers as a curl-like
Da> command.
I don't know if you can change the user agent in lynx, but try and see if it
works with curl.
e.g:
agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101
Firefox/127.0"
curl -A "$agent" <url>
I've had to do this in the past when pulling information for weather, they
stopped allowing curl, but simply changing the user agent allowed it to work.
Jay
... Modem: What landscapers do to dem lawns
--- Mystic BBS v1.12 A49 2024/05/29 (Linux/64)
* Origin: Northern Realms (1:229/664)
|
