Apache NiFi - Secure Connections to Apache Solr

Posted by Bryan Bende on October 13, 2017

This post is a tutorial showing how to setup a TLS/SSL connection from Apache NiFi to Solr.

Certificates

We can save ourselves some time and use NiFi’s TLS toolkit to generate everything we need.

Assuming you have downloaded the latest toolkit and extracted it somewhere, run the following command:

./bin/tls-toolkit.sh standalone -n 'localhost' -C 'CN=bbende, OU=NIFI' -o './target'

This should generate the following in the target directory where you ran the above command:

  target/
  ├── CN=bbende_OU=NIFI.p12
  ├── CN=bbende_OU=NIFI.password
  ├── localhost
  │   ├── keystore.jks
  │   ├── nifi.properties
  │   └── truststore.jks
  ├── nifi-cert.pem
  └── nifi-key.key
  • nifi-cert.pem - public certificate of the CA
  • nifi-key.key - private key of the CA
  • keystore.jks - public certificate and private key for CN=localhost, OU=NIFI
  • truststore.jks - public certificate of the CA (nifi-cert.pem)
  • nifi.properties - passwords for the keystore and truststore
  • CN=bbende_OU=NIFI.p12 - client certificate for your browser
  • CN=bbende_OU=NIFI.password - password for CN=bbende_OU=NIFI.p12

We can now configure Solr…

Solr config

  • Download the latest Solr release

    https://lucene.apache.org/solr/downloads.html

  • Extract the Solr release somewhere

    tar xzvf solr-7.0.1.tgz
cd solr-7.0.1

  • Copy the keystore and truststore from above

    mkdir certs
cp nifi-toolkit-home/target/localhost/* /path/to/solr-7.0.1/certs/

  • Edit bin/solr.in.sh to specify TLS/SSL properties

    SOLR_SSL_KEY_STORE=/path/to/solr-7.0.1/certs/keystore.jks
SOLR_SSL_KEY_STORE_PASSWORD=YOUR-PASSWORD
SOLR_SSL_KEY_STORE_TYPE=JKS
SOLR_SSL_TRUST_STORE=/path/to/solr/solr-7.0.1/certs/truststore.jks
SOLR_SSL_TRUST_STORE_PASSWORD=YOUR-PASSWORD
SOLR_SSL_TRUST_STORE_TYPE=JKS
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false

    The keystore and truststore passwords should be replaced with the real passwords which in our case are in nifi.properties in nifi.security.keystorePasswd and nifi.security.truststorePasswd respectively.

  • Start Solr

    ./bin/solr start

  • Create a core

    ./bin/solr create_core -c nifi-test

  • Verify Solr is up and running correctly over https with the created core

    https://localhost:8983/solr/#/nifi-test

    You will have to accept a security exception since we are using a self-signed certificate.

NiFi Config

For this post I am using Apache NiFi 1.4.0. I’m going to assume you can download and start NiFi.

  • Launch the NiFi UI in your browser

    http://localhost:8080/nifi/

  • Create an SSLContextService from context palette on the left and start the service

    NOTE: Since Solr currently has need-client-auth and want-client-auth set to false, we are only doing one-way TLS here, which means NiFi is verifying the certificate presented by Solr, but NiFi is not presenting a certificate itself.

  • Create a GenerateFlowFile processor to create some data

    Change the scheduling to run every 10 seconds, or something reasonable.

  • Create a PutSolrContentStream processor using the SSLContextService

  • Connect everything and start the flow

  • Check Solr Admin UI to see documents were ingested

What About Two-Way TLS/SSL?

  • Stop the flow in NiFi

  • Stop Solr

      /path/to/solr-7.0.1/bin/solr stop -all

  • Edit /path/to/solr-7.0.1/bin/solr.in.sh and require client auth

      SOLR_SSL_NEED_CLIENT_AUTH=true

  • Start Solr again

      /path/to/solr-7.0.1/bin/solr start

    NOTE: You will now need the client p12 from earlier loaded in your browser if you need to access Solr Admin UI

  • Add a keystore to the SSLContextService in NiFi

    NOTE: In a real environment, NiFi and Solr would likely be running on different machines, and each would have their own certificate in their own respective keystore. In this example, NiFi and Solr are both using the same keystore for simplicity.

  • Start the flow again and documents should be ingesting

What About Solr Cloud?

The Solr Reference Guide has really good documentation on enabling SSL, and has a specific section on how to enable SSL with Solr Cloud.

One thing to keep in mind is that the certificates you generate for Solr need to line-up with the hostname that Solr is binding to.

If you wanted to test this locally and use the same localhost cert we used above, you would need to tell each Solr instance to bind to localhost:

bin/solr -cloud -s cloud/node1 -z localhost:2181 -p 8984 -h localhost
bin/solr -cloud -s cloud/node2 -z localhost:2181 -p 7574 -h localhost

If you don’t do that, they will likely bind to an internal IP address which wont match the value in the certificate.

Another option for testing is to turn off peer verification as the Solr reference guide shows:

bin/solr -cloud -s cloud/node1 -z localhost:2181 -p 8984 -Dsolr.ssl.checkPeerName=false
bin/solr -cloud -s cloud/node2 -z localhost:2181 -p 7574 -Dsolr.ssl.checkPeerName=false

You should only turn off peer verification for testing.

On the NiFi side, you would have to reconfigure PutSolrContentStream to use cloud mode, and provide the ZooKeeper connection string, rather than the standalone URL.

blog comments powered by Disqus