[Sks-devel] Request: Install an efficient robots.txt file

[robots.txt fan]

Dear Sirs and Madams,

I would like to thank all of you for doing this. You are a necessary pillar to PGP and it is awesome that you are there to provide the infrastructure to host everyone's key.

Without attempting to diminish the previous sentence, I have a request to make to some of you.

Most of the SKS serve an efficient robots.txt that prevents everyone's un-deletable name and email showing up on search engines. However, there are some exceptions. I like to keep a low profile, but when searching for my name, for example on Google, a significant amount of results are from SKS pages, or to be more specific, these:

keyserver.nausch.org
pgp.net.nz
pgp.circl.lu
keyserver.rayservers.com
sks-keyservers.net
keyserver.mattrude.com (special case: blocks /pks, but not /search, a non-standard (?) directory)

I would like to ask the owners of these pages to take the time to install an efficient robots.txt file, for example something like this:

User-agent: *
Disallow: /pks/

To all others, I would like to ask you to take the time to check if your server serves an efficient robots.txt file, and if it does not, to please install one.

If there is any doubt that a robots.txt file is a good idea, I can elaborate on that.

Thank you for your time.

RTF

[Tobias Frei]

Hi,

If you don't want your name to appear on Google, don't upload it to a
service that permanently spreads it to hundreds of public websites.
Especially don't rely on every server admin to "block" crawlers from these
pages, because this fails as long as at least one admin doesn't.

Have a nice day anyway.

On Tue, Jun 20, 2017, 10:36 robots.txt fan <***@protonmail.com>
wrote:

> Dear Sirs and Madams,
>
> I would like to thank all of you for doing this. You are a necessary
> pillar to PGP and it is awesome that you are there to provide the
> infrastructure to host everyone's key.
>
> Without attempting to diminish the previous sentence, I have a request to
> make to some of you.
>
> Most of the SKS serve an efficient robots.txt that prevents everyone's
> un-deletable name and email showing up on search engines. However, there
> are some exceptions. I like to keep a low profile, but when searching for
> my name, for example on Google, a significant amount of results are from
> SKS pages, or to be more specific, these:
>
> keyserver.nausch.org
> pgp.net.nz
> pgp.circl.lu
> keyserver.rayservers.com
> sks-keyservers.net
> keyserver.mattrude.com (special case: blocks /pks, but not /search, a
> non-standard (?) directory)
>
> I would like to ask the owners of these pages to take the time to install
> an efficient robots.txt file, for example something like this:
>
> User-agent: *
> Disallow: /pks/
>
> To all others, I would like to ask you to take the time to check if your
> server serves an efficient robots.txt file, and if it does not, to please
> install one.
>
> If there is any doubt that a robots.txt file is a good idea, I can
> elaborate on that.
>
> Thank you for your time.
>
> RTF
> _______________________________________________
> Sks-devel mailing list
> Sks-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/sks-devel
>

[robots.txt fan]

Hi,

how can you assume that it was me who uploaded a key with my name on it?

Please, I try to be optimistic here. This is a problem where I rely on the server admins, yes. Unfixable for me, easily fixable for her or his server by the respective admin. Is it unreasonable to assume that the admins are benevolent? I do not think so. Otherwise, we may be talking about PBP, not PGP.

RTF

Hi,

If you don't want your name to appear on Google, don't upload it to a service that permanently spreads it to hundreds of public websites. Especially don't rely on every server admin to "block" crawlers from these pages, because this fails as long as at least one admin doesn't.

Have a nice day anyway.

On Tue, Jun 20, 2017, 10:36 robots.txt fan <***@protonmail.com> wrote:
Dear Sirs and Madams,

I would like to thank all of you for doing this. You are a necessary pillar to PGP and it is awesome that you are there to provide the infrastructure to host everyone's key.

Without attempting to diminish the previous sentence, I have a request to make to some of you.

Most of the SKS serve an efficient robots.txt that prevents everyone's un-deletable name and email showing up on search engines. However, there are some exceptions. I like to keep a low profile, but when searching for my name, for example on Google, a significant amount of results are from SKS pages, or to be more specific, these:

keyserver.nausch.org
pgp.net.nz
pgp.circl.lu
keyserver.rayservers.com
sks-keyservers.net
keyserver.mattrude.com (special case: blocks /pks, but not /search, a non-standard (?) directory)

I would like to ask the owners of these pages to take the time to install an efficient robots.txt file, for example something like this:

User-agent: *
Disallow: /pks/

To all others, I would like to ask you to take the time to check if your server serves an efficient robots.txt file, and if it does not, to please install one.

If there is any doubt that a robots.txt file is a good idea, I can elaborate on that.

Thank you for your time.

RTF

[Gabor Kiss]

On Tue, 20 Jun 2017, robots.txt fan wrote:

> From: robots.txt fan <***@protonmail.com>

> how can you assume that it was me who uploaded a key with my name on it?

Don't worry.
I searched your name (i.e. RTF) with Google and no hits came from
any key server on the fist five pages. :->

>
> Please, I try to be optimistic here. This is a problem where I rely on the server admins, yes. Unfixable for me, easily fixable for her or his server by the respective admin. Is it unreasonable to assume that the admins are benevolent? I do not think so. Otherwise, we may be talking about PBP, not PGP.
>
> RTF

Gabor

[Robert J. Hansen]

> how can you assume that it was me who uploaded a key with my name on it?

Nobody is. But if you create a public key, then by definition you're
comfortable with it being shared with the public.

If you don't want your public key shared with the public, don't use
asymmetric crypto.

If you didn't generate this key, then please accept my condolences on
some low-life jerk creating a key in your name with your email address
on it and uploading it to the keyservers. Those people are jerks.
Unfortunately, we have no good way to stop them.

[Ari Trachtenberg]

What about instituting an e-mail check before accepting a key with an e-mail?

> On Jun 20, 2017, at 8:26 AM, Robert J. Hansen <***@sixdemonbag.org> wrote:
>
> If you didn't generate this key, then please accept my condolences on
> some low-life jerk creating a key in your name with your email address
> on it and uploading it to the keyservers. Those people are jerks.
> Unfortunately, we have no good way to stop them.

—
Prof. Ari Trachtenberg
Electrical and Computer Engineering
Boston University
***@bu.edu

[Kristian Fiskerstrand]

On 06/20/2017 04:15 PM, Ari Trachtenberg wrote:
> What about instituting an e-mail check before accepting a key with an
> e-mail?

Then you're introducing an element of a certificate authority in the
wrong place (and not all public keyblocks have emails as UID to begin with).

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"If you choose to sail upon the seas of banking, build your bank as you
would your boat, with the strength to sail safely through any storm."
(Jacob Safra (1891–1963))

[Ari Trachtenberg]

Not quite ... each server can decide which keys it want s to accept.
Bad actors will eventually fall out of favor with the others.

> On Jun 20, 2017, at 10:21 AM, Kristian Fiskerstrand <***@sumptuouscapital.com> wrote:
>
> On 06/20/2017 04:15 PM, Ari Trachtenberg wrote:
>> What about instituting an e-mail check before accepting a key with an
>> e-mail?
>
> Then you're introducing an element of a certificate authority in the
> wrong place (and not all public keyblocks have emails as UID to begin with).
>
> --
> ----------------------------
> Kristian Fiskerstrand
> Blog: https://blog.sumptuouscapital.com
> Twitter: @krifisk
> ----------------------------
> Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
> fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
> ----------------------------
> "If you choose to sail upon the seas of banking, build your bank as you
> would your boat, with the strength to sail safely through any storm."
> (Jacob Safra (1891–1963))
>
> _______________________________________________
> Sks-devel mailing list
> Sks-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/sks-devel

—
Prof. Ari Trachtenberg
Electrical and Computer Engineering
Boston University
***@bu.edu

[Kristian Fiskerstrand]

On 06/20/2017 05:56 PM, Ari Trachtenberg wrote:
> Not quite ... each server can decide which keys it want s to accept.
> Bad actors will eventually fall out of favor with the others.

Now we presume a non-gossiping system of isolated servers

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Whenever you find yourself on the side of the majority, it is time to
pause and reflect."
(Mark Twain)

[Paul M Furley]

Hi RTF,

Thanks for your polite and reasonable request!

On 20/06/17 09:35, robots.txt fan wrote:
> Dear Sirs and Madams,
>
> I would like to thank all of you for doing this. You are a necessary
> pillar to PGP and it is awesome that you are there to provide the
> infrastructure to host everyone's key.
>
> Without attempting to diminish the previous sentence, I have a request
> to make to some of you.
>
> Most of the SKS serve an efficient robots.txt that prevents everyone's
> un-deletable name and email showing up on search engines.

It seems this is the default behaviour, at least when instaled from the
debian package.

It's always useful to hear from a real PGP user for whom this is a real
concern.

> However, there
> are some exceptions. I like to keep a low profile, but when searching
> for my name, for example on Google, a significant amount of results are
> from SKS pages, or to be more specific, these:
>
> keyserver.nausch.org
> pgp.net.nz
> pgp.circl.lu
> keyserver.rayservers.com
> sks-keyservers.net
> keyserver.mattrude.com (special case: blocks /pks, but not /search, a
> non-standard (?) directory)
>
> I would like to ask the owners of these pages to take the time to
> install an efficient robots.txt file, for example something like this:
>
> User-agent: *
> Disallow: /pks/
>
> To all others, I would like to ask you to take the time to check if your
> server serves an efficient robots.txt file, and if it does not, to
> please install one.
>
> If there is any doubt that a robots.txt file is a good idea, I can
> elaborate on that.
>
> Thank you for your time.
>
> RTF
>

My personal priority for the PGP ecosystem is to ensure a pleasant,
unsurprising and privacy respecting experience for all PGP users.

Since you've taken the time to email this list, I assume this is a
reasonably serious issue for you.

We should all be worried about anything that negatively affects the
overall experience of using PGP.

I also support your right to use PGP without publishing your public key.
It's not uncommon for users to have both published and non-published
keys, and we're here to support PGP users, so I'm behind you.

Probably too late now, but you might consider omitting your name
entirely on your key - it's optional. You don't even need an email
address, although that makes life a little tricky for your contacts ;)

Paul

>
> _______________________________________________
> Sks-devel mailing list
> Sks-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/sks-devel
>

[m***@ttwagner.com]

Hi,

Thanks for pointing this out. I never thought about robots.txt on my keyserver, and sure enough, it was missing.

It's easy to add: just drop it in the web/ directory in the skskeyserver directory. I did have to reload sks to get it to see the new file. It's now in place on gpg.n1zyy.com!

Beyond the privacy implications, I'm also happy to keep search engines from performing a ton of 'searches' on my server by following links between signed keys!

(And while it's true that _public_ keys are, well, public, I'm happy to not be directly giving Google et al. email addresses linked to real names.)

-----Original Message-----
From: "robots.txt fan" <***@protonmail.com>
Sent: Tuesday, June 20, 2017 4:35am
To: "sks-***@nongnu.org" <sks-***@nongnu.org>
Subject: [Sks-devel] Request: Install an efficient robots.txt file

_______________________________________________
Sks-devel mailing list
Sks-***@nongnu.org
https://lists.nongnu.org/mailman/listinfo/sks-devel
Dear Sirs and Madams,

I would like to thank all of you for doing this. You are a necessary pillar to PGP and it is awesome that you are there to provide the infrastructure to host everyone's key.

Without attempting to diminish the previous sentence, I have a request to make to some of you.

Most of the SKS serve an efficient robots.txt that prevents everyone's un-deletable name and email showing up on search engines. However, there are some exceptions. I like to keep a low profile, but when searching for my name, for example on Google, a significant amount of results are from SKS pages, or to be more specific, these:

keyserver.nausch.org
pgp.net.nz
pgp.circl.lu
keyserver.rayservers.com
sks-keyservers.net
keyserver.mattrude.com (special case: blocks /pks, but not /search, a non-standard (?) directory)

I would like to ask the owners of these pages to take the time to install an efficient robots.txt file, for example something like this:

User-agent: *
Disallow: /pks/

To all others, I would like to ask you to take the time to check if your server serves an efficient robots.txt file, and if it does not, to please install one.

If there is any doubt that a robots.txt file is a good idea, I can elaborate on that.

Thank you for your time.

RTF

[robots.txt fan]

Hello again,

m, thank you very much for installing the file to your server!

Paul, thank you for your kind words.

Robert, this is not a lost cause, but instead a fixable problem. Condolences are not required, but a solution is. This solution can only come from admins like m.

I have now come up with a larger list of servers that do not yet have an efficient robots.txt file or none at all. Kristian, you have responded to this thread, I believe you manage the first one on the list. Is there a reason why only /status is blocked and not /pks?

https://sks-keyservers.net (blocks /status, but not /pks)
https://keyserver.mattrude.com (blocks /pks, but not /search)
http://pgp.net.nz (works fine on port 11371, but not on port 80)
http://keyserver.nausch.org:11371 (completely missing)
http://pgp.circl.lu (completely missing)
http://keyserver.cns.vt.edu (completely missing)
https://gpg.mozilla.org (completely missing)
https://keyserver.metalgamer.eu (completely missing)
https://keys.fedoraproject.org (completely missing)
http://pgpkeys.eu:11371 (completely missing)
http://keyserver.rayservers.com:11371 (seems to be down now, was up a few days ago)

Best regards
RTF

[Kristian Fiskerstrand]

On 06/22/2017 10:40 AM, robots.txt fan wrote:
> Kristian, you have responded to this thread, I believe you manage the first one on the list. Is there a reason why only /status is blocked and not /pks?
>
> https://sks-keyservers.net (blocks /status, but not /pks)

The real reason is that /pks didn't exist when the robots.txt file was
created, so I've [added it now], granted more for site resource
management reasons than privacy reasons.

From a privacy perspective robots.txt doesn't make sense, the data is
already public, bad actors ignore robots.txt and crawl the site just the
same; and the full data set is available and part of regular workflow
for bootstrapping own servers.

References:
[added it now]
https://git.sumptuouscapital.com/?p=sks-keyservers-pool.git;a=commit;h=b98e7522990961541165dfc23781a45a1a5e05a9

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Whatever you do in life, surround yourself with smart people who'll
argue with you."
John Wooden

[Tobias Frei]

Hi robots.txt fan,

I just wondered what happened if I removed robots.txt from my server, to
reduce the whole discussion to absurdity. :)

Best regards,
ToBeFree

On Thu, Jun 22, 2017, 13:08 Kristian Fiskerstrand <
***@sumptuouscapital.com> wrote:

> On 06/22/2017 10:40 AM, robots.txt fan wrote:
> > Kristian, you have responded to this thread, I believe you manage the
> first one on the list. Is there a reason why only /status is blocked and
> not /pks?
> >
> > https://sks-keyservers.net (blocks /status, but not /pks)
>
> The real reason is that /pks didn't exist when the robots.txt file was
> created, so I've [added it now], granted more for site resource
> management reasons than privacy reasons.
>
> From a privacy perspective robots.txt doesn't make sense, the data is
> already public, bad actors ignore robots.txt and crawl the site just the
> same; and the full data set is available and part of regular workflow
> for bootstrapping own servers.
>
> References:
> [added it now]
>
> https://git.sumptuouscapital.com/?p=sks-keyservers-pool.git;a=commit;h=b98e7522990961541165dfc23781a45a1a5e05a9
>
> --
> ----------------------------
> Kristian Fiskerstrand
> Blog: https://blog.sumptuouscapital.com
> Twitter: @krifisk
> ----------------------------
> Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
> fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
> ----------------------------
> "Whatever you do in life, surround yourself with smart people who'll
> argue with you."
> John Wooden
>
> _______________________________________________
> Sks-devel mailing list
> Sks-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/sks-devel
>

[robots.txt fan]

metalgamer: Thank you very much!

ToBeFree: It would sure serve the absurdity indeed. Please don't do it.

Kristian: Thank you very much for adding the file to the repository! Like I explained, the concern are not bad actors here, but instead actors that do respect the standard (e.g. Google). May I ask how the git repository and the live site are related? While I see the robots.txt file in the git repository, it is not displayed on https://sks-keyservers.net/robots.txt.

Best regards
RTF

[Kristian Fiskerstrand]

On 06/23/2017 11:10 AM, robots.txt fan wrote:
> metalgamer: Thank you very much!
>
> ToBeFree: It would sure serve the absurdity indeed. Please don't do
> it.
>
> Kristian: Thank you very much for adding the file to the repository!
> Like I explained, the concern are not bad actors here, but instead
> actors that do respect the standard (e.g. Google). May I ask how the
> git repository and the live site are related? While I see the
> robots.txt file in the git repository, it is not displayed on
> https://sks-keyservers.net/robots.txt.
>

Thank you for heads up, given that robots.txt wasn't previously tracked
but created directly on server there ended up a conflict on update for
the file...

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Money is better than poverty, if only for financial reasons."
(Woody Allen)

[robots.txt fan]

But now it is working, thank you very much!

> Thank you for heads up, given that robots.txt wasn"t previously tracked
> but created directly on server there ended up a conflict on update for
> the file...

[Dennis Fink]

> https://keyserver.metalgamer.eu (completely missing)

Done.

Cheers,
metalgamer

[Daniel Austin]

Hi,

On 22/06/2017 09:40, robots.txt fan wrote:
> http://pgpkeys.eu:11371 (completely missing)

Whilst I don't believe it will make any difference whatsoever to your
spam levels, it may reduce some load on my keyservers from genuine
indexing so I've added a robots.txt file at the root (covering both port
11371 and 80).

This has been applied to:

pgpkeys.eu
pgpkeys.co.uk



Thanks,

Daniel.

[robots.txt fan]

It is not about spam, but about being found. Thank you very much for adding the file!
Only 5 or 6 of the servers I found are left.

>> Whilst I don"t believe it will make any difference whatsoever to your
>> spam levels, it may reduce some load on my keyservers from genuine
>> indexing so I"ve added a robots.txt file at the root (covering both port
>> 11371 and 80).