Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node key not being exported to pkcs8 #294

Open
jacobreid opened this issue Mar 18, 2019 · 0 comments
Open

Node key not being exported to pkcs8 #294

jacobreid opened this issue Mar 18, 2019 · 0 comments

Comments

@jacobreid
Copy link

I can create a cluster with the upmcenterprises/docker-elasticsearch-kubernetes:6.1.3_1 image fine, but when I try to use the searchguard image (https://github.com/while1eq1/elasticsearch-kubernetes-searchguard), elasticsearch fails to start up because the node key can not be found. The searchguard config specifies this to be in pkcs8 format (https://github.com/while1eq1/elasticsearch-kubernetes-searchguard/blob/master/config/elasticsearch.yml#L47) and this should be written out (https://github.com/upmc-enterprises/elasticsearch-operator/blob/master/pkg/k8sutil/certs.go#L206)

Changing ownership of /elasticsearch folder
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/truststore.jks: Read-only file system
Changing ownership of /data folder
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/truststore.jks: Read-only file system
Waiting for Elasticsearch to become ready before running sgadmin...
[2019-03-18T15:47:18,803][INFO ][o.e.n.Node               ] [aae567bf-aa89-4558-b2fe-7c78083abd99] initializing ...
[2019-03-18T15:47:18,862][INFO ][o.e.e.NodeEnvironment    ] [aae567bf-aa89-4558-b2fe-7c78083abd99] using [1] data paths, mounts [[/data (/dev/nvme0n1p2)]], net usable_space [89.7gb], net total_space [119.9gb], types [ext4]
[2019-03-18T15:47:18,862][INFO ][o.e.e.NodeEnvironment    ] [aae567bf-aa89-4558-b2fe-7c78083abd99] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-03-18T15:47:18,863][INFO ][o.e.n.Node               ] [aae567bf-aa89-4558-b2fe-7c78083abd99] node name [aae567bf-aa89-4558-b2fe-7c78083abd99], node ID [cbnqrXMlT66u6oE927Y0GA]
[2019-03-18T15:47:18,863][INFO ][o.e.n.Node               ] [aae567bf-aa89-4558-b2fe-7c78083abd99] version[6.4.1], pid[19], build[default/tar/e36acdb/2018-09-13T22:18:07.696808Z], OS[Linux/4.9.0-7-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-03-18T15:47:18,863][INFO ][o.e.n.Node               ] [aae567bf-aa89-4558-b2fe-7c78083abd99] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms1024m, -Xmx1024m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-03-18T15:47:20,262][INFO ][o.e.p.p.PrometheusExporterPlugin] starting Prometheus exporter plugin
[2019-03-18T15:47:20,405][INFO ][c.f.s.SearchGuardPlugin  ] ES Config path is /elasticsearch/config
[2019-03-18T15:47:20,447][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2019-03-18T15:47:20,454][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively
[2019-03-18T15:47:20,508][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [aae567bf-aa89-4558-b2fe-7c78083abd99] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
	... 6 more
Caused by: org.elasticsearch.ElasticsearchException: Unable to read /elasticsearch/config/certs/node-key.pkcs8.pem (/elasticsearch/config/certs/node-key.pkcs8.pem). Please make sure this files exists and is readable regarding to permissions. Property: searchguard.ssl.transport.pemkey_filepath
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:809) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:210) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:327) ~[?:?]
	at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
	at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
	at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
	... 6 more
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...

On one of the nodes:

bash-4.4# ls /elasticsearch/config/certs/
ca-key.pem         cerebro-key.pem    kibana-key.pem     node-key.pem       node.pem
ca.pem             cerebro.pem        kibana.pem         node-keystore.jks  truststore.jks
bash-4.4#

If there is some configuration required for the key to be exported to pkcs8 this should be documented in the README.

Using elasticsearch-operator 0.3.0 on Kubernetes v1.10.11.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jacobreid