[Sks-devel] setting up hockeypuck keyserver

Discussion:

Fabian A. Santiago

2018-08-31 21:06:17 UTC

can anyone on here assist with setting up hockeypuck?

I've received help prior from a friendly user who got me up & running
with mongodb. but it was SLOW to import a dump (days) and never caught
up with my sks keyserver (maybe erroring out on gossip).

would it be faster using postgresql?

is there something special to configure in order to get it to gossip
properly with sks servers as well as other hockeypucks?

are there any instructions out there for setting up postgresql properly
to be used with it?

anything anyone can provide would be great as the hockeypuck docs site
https://hockeypuck.github.io doesn't contain much in the way of finer
details.

thanks.
--
Fabian S.

OpenPGP:

0x643082042DC83E6D94B86C405E3DAA18A1C22D8F (new key)

***

0x3C3FA072ACCB7AC5DB0F723455502B0EEB9070FC (to be retired, still valid)

dirk astrath

2018-09-01 06:14:02 UTC

Permalink

Hello,

Postgres ist way faster and sustainable compared to mongodb (tested both
on different machines).

The recon-parameters on hockeypuck-site needs to be modified so it can
synchronize to the network (check the logfiles on sks and hockeypuck size).

Unfortunately I don't run any keyserver currently due to several issues,
so i need to pull these settings out of a backup ... but can do it if
neccessary ...

Kind regards,

dirk

Post by Fabian A. Santiago
can anyone on here assist with setting up hockeypuck?
I've received help prior from a friendly user who got me up & running
with mongodb. but it was SLOW to import a dump (days) and never caught
up with my sks keyserver (maybe erroring out on gossip).
would it be faster using postgresql?
is there something special to configure in order to get it to gossip
properly with sks servers as well as other hockeypucks?
are there any instructions out there for setting up postgresql properly
to be used with it?
anything anyone can provide would be great as the hockeypuck docs site
https://hockeypuck.github.io doesn't contain much in the way of finer
details.
thanks.
_______________________________________________
Sks-devel mailing list
https://lists.nongnu.org/mailman/listinfo/sks-devel

Fabian A. Santiago

2018-09-01 23:10:02 UTC

Permalink

Could definitely use help on the parameters. And yes it was erorring on recon. So that explains why it never caught up. Iâd appreciate the config help if you could. Thanks.

--

Post by dirk astrath
Hello,
Postgres ist way faster and sustainable compared to mongodb (tested both
on different machines).
The recon-parameters on hockeypuck-site needs to be modified so it can
synchronize to the network (check the logfiles on sks and hockeypuck size).
Unfortunately I don't run any keyserver currently due to several issues,
so i need to pull these settings out of a backup ... but can do it if
neccessary ...
Kind regards,
dirk

Moritz Wirth

2018-09-01 23:14:26 UTC

Permalink

You have to set the filters to get the reconciliation working with SKS:

[hockeypuck.conflux.recon]
reconAddr=":11370"
version="1.1.6"
filters=["yminsky.dedup", "yminsky.merge"]

For Postgresql:

[hockeypuck.openpgp.db]
driver="postgres-jsonb"
dsn="database=hkp host=/var/run/postgresql port=5432 sslmode=disable"

Create the database and run hockeypuck-load as postgre user - that should be enough to start with a clean installation (you must use postgresql > 9.4) - the tables are created by hockeypuck-load - 100k keys take about 3-5 minutes to be imported so the whole process should be done in 3-4 hours.

In case your import fails, I can provide a database dump of my MongoDB/PostgreSQL databases.

Regards,

Moritz

Post by Fabian A. Santiago
Could definitely use help on the parameters. And yes it was erorring
on recon. So that explains why it never caught up. Id appreciate the
config help if you could. Thanks.
--
On Saturday, Sep 01, 2018 at 2:14 AM, dirk astrath
Hello,
Postgres ist way faster and sustainable compared to mongodb (tested both
on different machines).
The recon-parameters on hockeypuck-site needs to be modified so it can
synchronize to the network (check the logfiles on sks and
hockeypuck size).
Unfortunately I don't run any keyserver currently due to several issues,
so i need to pull these settings out of a backup ... but can do it if
neccessary ...
Kind regards,
dirk

_______________________________________________
Sks-devel mailing list
https://lists.nongnu.org/mailman/listinfo/sks-devel

Fabian A. Santiago

2018-09-02 01:42:17 UTC

Permalink

Close,

Command needs to be run as hockeypuck user.

You need to create an hkp dB and hockeypuck role and grant ownership over the dB (at least that worked for me).

Otherwise, itâs working and importing now. Thanks for your help. Iâll advise when itâs done and I can see whether or not I have good recon.

--

Post by Moritz Wirth
[hockeypuck.conflux.recon]
reconAddr=":11370"
version="1.1.6"
filters=["yminsky.dedup", "yminsky.merge"]
[hockeypuck.openpgp.db] driver="postgres-jsonb" dsn="database=hkp host=/var/run/postgresql port=5432 sslmode=disable" Create the database and run hockeypuck-load as postgre user - that should be enough to start with a clean installation (you must use postgresql > 9.4) - the tables are created by hockeypuck-load - 100k keys take about 3-5 minutes to be imported so the whole process should be done in 3-4 hours. In case your import fails, I can provide a database dump of my MongoDB/PostgreSQL databases. Regards, Moritz

Post by Fabian A. Santiago
Could definitely use help on the parameters. And yes it was erorring on recon. So that explains why it never caught up. Iâd appreciate the config help if you could. Thanks.
--

Fabian A. Santiago

2018-09-02 22:54:06 UTC

Permalink

Alright....import was much quicker. But now the service won't start. Triaging now.

--

Post by Fabian A. Santiago
Close,
Command needs to be run as hockeypuck user.
You need to create an hkp dB and hockeypuck role and grant ownership over the dB (at least that worked for me).
Otherwise, itâs working and importing now. Thanks for your help. Iâll advise when itâs done and I can see whether or not I have good recon.
--

Fabian A. Santiago

2018-09-02 23:08:40 UTC

Permalink

Ok,

So trying to run in user space produces:

[{/build/hockeypuck-lsqW7K/hockeypuck-2.0~rc2~1437340214+54df397~trusty/src/gopkg.in/hockeypuck/pghkp.v1/storage.go:112: } {/build/hockeypuck-lsqW7K/hockeypuck-2.0~rc2~1437340214+54df397~trusty/src/gopkg.in/hockeypuck/pghkp.v1/storage.go:122: } {dial unix /var/run/postgresql/.s.PGSQL.5432: no such file or directory}]

I have postgresql version 10.5 installed on Ubuntu 14.04.5. Thanks in advance for any help you can offer.

--

Post by Fabian A. Santiago
Alright....import was much quicker. But now the service won't start. Triaging now.
--

Fabian A. Santiago

2018-09-02 23:15:55 UTC

Permalink

I should add that it worked fine using Mongodb but was so slow. If I hadn't made that clear prior.

--

Post by Fabian A. Santiago
Ok,
[{/build/hockeypuck-lsqW7K/hockeypuck-2.0~rc2~1437340214+54df397~trusty/src/gopkg.in/hockeypuck/pghkp.v1/storage.go:112: } {/build/hockeypuck-lsqW7K/hockeypuck-2.0~rc2~1437340214+54df397~trusty/src/gopkg.in/hockeypuck/pghkp.v1/storage.go:122: } {dial unix /var/run/postgresql/.s.PGSQL.5432: no such file or directory}]
I have postgresql version 10.5 installed on Ubuntu 14.04.5. Thanks in advance for any help you can offer.
--

Post by Fabian A. Santiago
Alright....import was much quicker. But now the service won't start. Triaging now.
--

Moritz Wirth

2018-09-02 23:27:27 UTC

Permalink

Looks like PostgreSQL is not running or you got the wrong path to the
unix socket.

Post by Fabian A. Santiago
I should add that it worked fine using Mongodb but was so slow. If I
hadn't made that clear prior.
--
On Sunday, Sep 02, 2018 at 7:08 PM, Fabian A. Santiago
Ok,
}
} {dial unix /var/run/postgresql/.s.PGSQL.5432: no such file or
directory}]
I have postgresql version 10.5 installed on Ubuntu 14.04.5. Thanks
in advance for any help you can offer.
--
On Sunday, Sep 02, 2018 at 6:54 PM, Fabian A. Santiago
Alright....import was much quicker. But now the service won't
start. Triaging now.
--
On Saturday, Sep 01, 2018 at 9:42 PM, Fabian A. Santiago
Close,
Command needs to be run as hockeypuck user.
You need to create an hkp dB and hockeypuck role and grant
ownership over the dB (at least that worked for me).
Otherwise, its working and importing now. Thanks for your
help. Ill advise when its done and I can see whether or
not I have good recon.
--
On Saturday, Sep 01, 2018 at 7:14 PM, Moritz Wirth
You have to set the filters to get the reconciliation
[hockeypuck.conflux.recon]
reconAddr=":11370"
version="1.1.6"
filters=["yminsky.dedup", "yminsky.merge"]
[hockeypuck.openpgp.db] driver="postgres-jsonb" dsn="database=hkp host=/var/run/postgresql port=5432 sslmode=disable" Create the database and run hockeypuck-load as postgre user - that should be enough to start with a clean installation (you must use postgresql > 9.4) - the tables are created by hockeypuck-load - 100k keys take about 3-5 minutes to be imported so the whole process should be done in 3-4 hours. In case your import fails, I can provide a database dump of my MongoDB/PostgreSQL databases. Regards, Moritz

Post by Fabian A. Santiago
Could definitely use help on the parameters. And yes
it was erorring on recon. So that explains why it
never caught up. Id appreciate the config help if
you could. Thanks.
--
On Saturday, Sep 01, 2018 at 2:14 AM, dirk
Hello,
Postgres ist way faster and sustainable compared
to mongodb (tested both
on different machines).
The recon-parameters on hockeypuck-site needs to
be modified so it can
synchronize to the network (check the logfiles on
sks and hockeypuck size).
Unfortunately I don't run any keyserver currently
due to several issues,
so i need to pull these settings out of a backup
... but can do it if
neccessary ...
Kind regards,
dirk

Post by Fabian A. Santiago
can anyone on here assist with setting up
hockeypuck?
I've received help prior from a friendly user
who got me up & running
with mongodb. but it was SLOW to import a dump
(days) and never caught
up with my sks keyserver (maybe erroring out on
gossip).
would it be faster using postgresql?
is there something special to configure in order
to get it to gossip
properly with sks servers as well as other
hockeypucks?
are there any instructions out there for setting
up postgresql properly
to be used with it?
anything anyone can provide would be great as
the hockeypuck docs site
https://hockeypuck.github.io doesn't contain
much in the way of finer
details.
thanks.
_______________________________________________
Sks-devel mailing list
https://lists.nongnu.org/mailman/listinfo/sks-devel

_______________________________________________
Sks-devel mailing list
https://lists.nongnu.org/mailman/listinfo/sks-devel

Fabian A. Santiago

2018-09-03 00:17:36 UTC

Permalink

Ahh, damn, it had stopped. Ok service starts now but Iâm missing keys which suggests it stopped during the key load for which I think I may know why. Ok back to the drawing board. Standby. Thanks.

--

Fabian A. Santiago

OpenPGP:

0x643082042dc83e6d94b86c405e3daa18a1c22d8f

Looks like PostgreSQL is not running or you got the wrong path to the unix socket.

Post by Fabian A. Santiago
I should add that it worked fine using Mongodb but was so slow. If I hadn't made that clear prior.
--

Post by Fabian A. Santiago
Alright....import was much quicker. But now the service won't start. Triaging now.
--

_______________________________________________
Sks-devel mailing list
https://lists.nongnu.org/mailman/listinfo/sks-devel

Fabian A. Santiago

2018-09-04 21:14:55 UTC

Permalink

Yeh it had stopped cause I ran out of disk space during the key import. Resizing now and will try again later.

--

Fabian A. Santiago

OpenPGP:

0x643082042dc83e6d94b86c405e3daa18a1c22d8f

Post by Fabian A. Santiago
Ahh, damn, it had stopped. Ok service starts now but Iâm missing keys which suggests it stopped during the key load for which I think I may know why. Ok back to the drawing board. Standby. Thanks.
--
Fabian A. Santiago
0x643082042dc83e6d94b86c405e3daa18a1c22d8f

Looks like PostgreSQL is not running or you got the wrong path to the unix socket.

Post by Fabian A. Santiago
I should add that it worked fine using Mongodb but was so slow. If I hadn't made that clear prior.
--

Post by Fabian A. Santiago
Alright....import was much quicker. But now the service won't start. Triaging now.
--

_______________________________________________
Sks-devel mailing list
https://lists.nongnu.org/mailman/listinfo/sks-devel

Fabian A. Santiago

2018-09-06 11:16:45 UTC

Permalink

Hot Dog, finally itâs fully up and running with the correct # of expected keys. And syncing with my sks server (hopefully). Will check the logs out in a bit. Thanks for your help everyone. Ideally I think I would like to move my peers into the hockeypuck server eventually assuming I see its working with my sks and gossiping ok. Standby.

--

Fabian A. Santiago

OpenPGP:

0x643082042dc83e6d94b86c405e3daa18a1c22d8f

Post by Fabian A. Santiago
Yeh it had stopped cause I ran out of disk space during the key import. Resizing now and will try again later.
--
Fabian A. Santiago
0x643082042dc83e6d94b86c405e3daa18a1c22d8f

Looks like PostgreSQL is not running or you got the wrong path to the unix socket.

Post by Fabian A. Santiago
I should add that it worked fine using Mongodb but was so slow. If I hadn't made that clear prior.
--

Post by Fabian A. Santiago
Alright....import was much quicker. But now the service won't start. Triaging now.
--

_______________________________________________
Sks-devel mailing list
https://lists.nongnu.org/mailman/listinfo/sks-devel

Andrew Gallagher

2018-09-06 11:56:32 UTC

Permalink

Hot Dog, finally its fully up and running with the correct # of
expected keys. And syncing with my sks server (hopefully). Will check
the logs out in a bit. Thanks for your help everyone. Ideally I think I
would like to move my peers into the hockeypuck server eventually
assuming I see its working with my sks and gossiping ok. Standby.

I for one would be most interested if you could collate your experiences
into a howto... :-)

--
Andrew Gallagher

Fabian A. Santiago

2018-09-06 13:39:06 UTC

Permalink

Post by Andrew Gallagher
I for one would be most interested if you could collate your
experiences
into a howto... :-)

i will gladly write something up about my experience once i'm sure it's
up and running properly.

looking into my recon logs on both ends (hockeypuck and my prior sks
server), i question some of what i see.

in sks recon.log i see sks requesting missing keys from h..puck and
getting them. but i also see sks reporting "recon attempt FROM h..puck
while gossip disabled. ignoring." so i'm not sure but it seems something
is still off with recon.

on the h..puck side i see in its log that SOME connections from sks have
been reset by peer. so these i'm assuming are the recon ignores reported
by sks.

when i look at both the sks pool status pages on the web and my own
h..puck's status page, the h..puck server's version number is listed as
2.0~rc2~..., even though in my config file i have set the
'version="1.1.6"' parameter. so i also don't know why it's misreporting
its version, apparently. any thoughts? or have i misunderstood what the
field should say? i also have in my h..puck config two filters,
yminsky.dedup & yminsky.merge. i did notice on my h..puck stats page,
under settings, my http is reported as 127.0.0.1:11371 (I'm reverse
proxied behind nginx) and recon as simply :11370. does this seem
correct? you can see it for yourself at:

http://keys2.fspproductions.biz/pks/lookup?op=stats

does anyone have a working hockeypuck server they care to peer with my
own? My key count sits at 5312283. My peer line for hockeypuck is:

[hockeypuck.conflux.recon.partner.peerX]
httpAddr="keys2.fspproductions.biz:11371"
reconAddr="keys2.fspproductions.biz:11370"

and my contact remains

Fabian A. Santiago <***@garbage-juice.com>
0x643082042dc83e6d94b86c405e3daa18a1c22d8f

The server should be reachable via both IPv4 & 6 but currently does not
offer ssl (hkps). It will eventually once I'm done setting it up but NOT
thru an official pool trusted cert.

Thanks.

--
Fabian S.

OpenPGP:

0x643082042DC83E6D94B86C405E3DAA18A1C22D8F

Fabian A. Santiago

2018-09-06 14:32:21 UTC

Permalink

Post by Fabian A. Santiago

Post by Andrew Gallagher
I for one would be most interested if you could collate your
experiences
into a howto... :-)

i will gladly write something up about my experience once i'm sure
it's up and running properly.
looking into my recon logs on both ends (hockeypuck and my prior sks
server), i question some of what i see.
in sks recon.log i see sks requesting missing keys from h..puck and
getting them. but i also see sks reporting "recon attempt FROM h..puck
while gossip disabled. ignoring." so i'm not sure but it seems
something is still off with recon.
on the h..puck side i see in its log that SOME connections from sks
have been reset by peer. so these i'm assuming are the recon ignores
reported by sks.
when i look at both the sks pool status pages on the web and my own
h..puck's status page, the h..puck server's version number is listed
as 2.0~rc2~..., even though in my config file i have set the
'version="1.1.6"' parameter. so i also don't know why it's
misreporting its version, apparently. any thoughts? or have i
misunderstood what the field should say? i also have in my h..puck
config two filters, yminsky.dedup & yminsky.merge. i did notice on my
h..puck stats page, under settings, my http is reported as
127.0.0.1:11371 (I'm reverse proxied behind nginx) and recon as simply
http://keys2.fspproductions.biz/pks/lookup?op=stats
does anyone have a working hockeypuck server they care to peer with my
[hockeypuck.conflux.recon.partner.peerX]
httpAddr="keys2.fspproductions.biz:11371"
reconAddr="keys2.fspproductions.biz:11370"
and my contact remains
0x643082042dc83e6d94b86c405e3daa18a1c22d8f
The server should be reachable via both IPv4 & 6 but currently does
not offer ssl (hkps). It will eventually once I'm done setting it up
but NOT thru an official pool trusted cert.
Thanks.

i can also add that my key count is steadily rising. i'm now at 5312329,
so it appears as though it may be working even though i don't fully
understand why or what i'm seeing in the logs.

--
Fabian S.

OpenPGP:

0x643082042DC83E6D94B86C405E3DAA18A1C22D8F