[bitcoindev] Broken links to the previous mailing list archive

[bitcoindev] Broken links to the previous mailing list archive

[Bryan Bishop]

Hi,

lists.linuxfoundation.org is no longer hosting the static HTML email
archives for bitcoin-dev or any other mailing list. We knew that this
might happen and prepared months ago with the mailing list migration
and also with various backups:
https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/

However, many thousands of links around the web are still pointing to
the old archive. This is really unfortunate as there is a wealth of
content in the archive and these links were considered canonical. This
is a general plea to update any broken links that you might be able to
fix.

What you can do, options
========================

There are a few options for actions that effected content providers can take:

1) Use `gnusha.org/url`. Explained below.

2) Replace lists.linuxfoundation.org urls with resolved urls to a new
location for mailing list archives. More on this below.

3) Do nothing. Wait and see. It is possible that we can plead our case
to Linux Foundation and get them to host static archives indefinitely,
or at least host redirects.

4) Replace broken links with Wayback Machine urls. Someone told me
apparently their archive is incomplete? I don't recommend this one
because I am concerned about stress and strain on the Internet Archive
Wayback Machine service beyond the additional activity that these
links would incur. Besides, Wayback is already a highly centralized
point for internet archival activity and therefore some
decenteralization is warranted here.


gnusha.org/url redirect service
===============================

This is a redirect service I made that takes a URL parameter and
redirects. To use it, get a previous url like:

`https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html`

your new url will be:

`https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html`

and it will redirect to a current mailing list archive location for
that message:

`https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1@blaine.gmane.org/`

It uses a mapping file based on the mailing list archives. I have
published scripts to make this mapping here:
https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b

See https://gnusha.org/url for more information. Thank you to Jonas
Schnelli for suggesting the creation of this redirect service.


Manual resolution
=================

How to "manually" resolve "without" the gnusha.org/url redirect
service. Use resolver.py here:
https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b#file-resolver-py
and set RESOLVER_HOST to `https://gnusha.org/url`, also set `resolver
= resolve_locally` to `resolver = resolve_remotely`.

This will convert urls from the old LF format to resolved urls like:
`https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/`
or
`https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/`
depending on which public-inbox mirror you choose to use.


Why resolve or not
==================

I'll elaborate on `gnusha.org/url` vs `gnusha.org/pi/bitcoindev` style urls.

It is nice and simple to use a find-and-replace with `gnusha.org/url/`
because anyone can easily repeat that exercise and verify the git diff
before merging the pull request. However, the actual routing is not
something that is verified by just checking that the `git diff` looks
sane.

If I was to provide my mapping (I haven't uploaded this yet, but I did
publish the scripts I used to make it, hopefully someone will
re-generate it to verify my work), then someone could write a script
and/or cross-verify that all of the replacements are valid according
to the mapping, if we were to use the destination urls instead of the
redirect service for this pull request. Then someone would be able to
verify that kind of pull request.

There are other further issues though, like content malleability,
wrong redirect link injection, service provider compromise... instead
of referencing by Message-ID maybe in the future we can reference by
email hash.... But now I'm dreaming.



- Bryan
https://x.com/kanzure

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/CABaSBaz13bUoHCupXYhmX%2ByS0dn89f80yx8ZD3uO5-1RiLZJCQ%40mail.gmail.com.

[bitcoindev] Re: Broken links to the previous mailing list archive

[Weikeng Chen]

[-- Attachment #1.1: Type: text/plain, Size: 6436 bytes --]

I have two points.

(1) I think it is a good idea for the Bitcoin mail lists to purchase a 
domain of its own in the future because it is hard to expect any 
open-source foundation to long live Bitcoin. 

(2) From the previous 
mail https://gnusha.org/pi/bitcoindev/CABaSBaz9OTSVa1KNk0GOrH3T-kRF_7OPVu0AtpuaFGVB=zhdwQ@mail.gmail.com/, 
it seems that Linux Foundation may be happy to do some redirects, maybe it 
could be coordinated?

"Linux Foundation has either offered or agreed to maintain archive 
permalinks so that content of historic importance is not lost. Fortunately 
for us while lists.linuxfoundation.org mailman will go down, they have 
agreed the read-only pipermail archives will remain online."

Weikeng
On Wednesday, November 13, 2024 at 2:55:58 AM UTC+7 Bryan Bishop wrote:

> Hi,
>
> lists.linuxfoundation.org is no longer hosting the static HTML email
> archives for bitcoin-dev or any other mailing list. We knew that this
> might happen and prepared months ago with the mailing list migration
> and also with various backups:
>
> https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_Z...@mail.gmail.com/ 
> <https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/>
>
> However, many thousands of links around the web are still pointing to
> the old archive. This is really unfortunate as there is a wealth of
> content in the archive and these links were considered canonical. This
> is a general plea to update any broken links that you might be able to
> fix.
>
> What you can do, options
> ========================
>
> There are a few options for actions that effected content providers can 
> take:
>
> 1) Use `gnusha.org/url` <http://gnusha.org/url>. Explained below.
>
> 2) Replace lists.linuxfoundation.org urls with resolved urls to a new
> location for mailing list archives. More on this below.
>
> 3) Do nothing. Wait and see. It is possible that we can plead our case
> to Linux Foundation and get them to host static archives indefinitely,
> or at least host redirects.
>
> 4) Replace broken links with Wayback Machine urls. Someone told me
> apparently their archive is incomplete? I don't recommend this one
> because I am concerned about stress and strain on the Internet Archive
> Wayback Machine service beyond the additional activity that these
> links would incur. Besides, Wayback is already a highly centralized
> point for internet archival activity and therefore some
> decenteralization is warranted here.
>
>
> gnusha.org/url redirect service
> ===============================
>
> This is a redirect service I made that takes a URL parameter and
> redirects. To use it, get a previous url like:
>
> `
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html` 
> <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html>
>
> your new url will be:
>
> `
> https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html` 
> <https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html>
>
> and it will redirect to a current mailing list archive location for
> that message:
>
> `https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1...@blaine.gmane.org/` 
> <https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1@blaine.gmane.org/>
>
> It uses a mapping file based on the mailing list archives. I have
> published scripts to make this mapping here:
> https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b
>
> See https://gnusha.org/url for more information. Thank you to Jonas
> Schnelli for suggesting the creation of this redirect service.
>
>
> Manual resolution
> =================
>
> How to "manually" resolve "without" the gnusha.org/url redirect
> service. Use resolver.py here:
>
> https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b#file-resolver-py
> and set RESOLVER_HOST to `https://gnusha.org/url` <https://gnusha.org/url>, 
> also set `resolver
> = resolve_locally` to `resolver = resolve_remotely`.
>
> This will convert urls from the old LF format to resolved urls like:
> `
> https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_Z...@mail.gmail.com/` 
> <https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/>
> or
> `
> https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_Z...@mail.gmail.com/` 
> <https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/>
> depending on which public-inbox mirror you choose to use.
>
>
> Why resolve or not
> ==================
>
> I'll elaborate on `gnusha.org/url` <http://gnusha.org/url> vs `
> gnusha.org/pi/bitcoindev` <http://gnusha.org/pi/bitcoindev> style urls.
>
> It is nice and simple to use a find-and-replace with `gnusha.org/url/` 
> <http://gnusha.org/url/>
> because anyone can easily repeat that exercise and verify the git diff
> before merging the pull request. However, the actual routing is not
> something that is verified by just checking that the `git diff` looks
> sane.
>
> If I was to provide my mapping (I haven't uploaded this yet, but I did
> publish the scripts I used to make it, hopefully someone will
> re-generate it to verify my work), then someone could write a script
> and/or cross-verify that all of the replacements are valid according
> to the mapping, if we were to use the destination urls instead of the
> redirect service for this pull request. Then someone would be able to
> verify that kind of pull request.
>
> There are other further issues though, like content malleability,
> wrong redirect link injection, service provider compromise... instead
> of referencing by Message-ID maybe in the future we can reference by
> email hash.... But now I'm dreaming.
>
>
>
> - Bryan
> https://x.com/kanzure
>

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/b2f2dadc-97b1-4333-8916-426292d5ec8cn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 12124 bytes --]

Re: [bitcoindev] Re: Broken links to the previous mailing list archive

[Andrew Poelstra]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

On Tue, Nov 12, 2024 at 06:35:31PM -0800, Weikeng Chen wrote:
> 
> (1) I think it is a good idea for the Bitcoin mail lists to purchase a 
> domain of its own in the future because it is hard to expect any 
> open-source foundation to long live Bitcoin. 
> 

Unfortunately there is no "Bitcoin mailing list" entity that could
purchase a domain or hosting on its own. It would be possible to set one
up but ultimately it'd ground out in community members volunteering
their own resources to maintain the archive. (This is exactly what is
happening with the new gnusha.org archives.)

I suppose in theory somebody with a lot of cash could set up an org with
a charter and an endowment that could pay some sysadmin forever...but
it's hard to see anybody stepping up for the sake of email archives.
And as we've seen from our own problems with the Linux Foundation
servers, and from the recent hack on the Internet Archive, that even
this isn't likely to be sufficient.

The hope by using the LF servers was that, because they are a large and
diversified organization, and ideologically aligned with providing
access to information, that they would stick around. It's pretty
frustrating and disappointing that this isn't the case.


-- 
Andrew Poelstra
Director, Blockstream Research
Email: apoelstra at wpsoftware.net
Web:   https://www.wpsoftware.net/andrew

The sun is always shining in space
    -Justin Lewis-Webster

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/ZzYJejfeHC4RcSRG%40camus.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]