Custom Processors for Apache NiFi

Posted by Bryan Bende on February 4, 2015

This post will cover the initial steps to get started developing a custom processor for Apache NiFi. The focus will be on setting up a new processor bundle and and deploying it.

The specifics of implementing a processor can be found in the NiFi Developer’s Guide, as well as many examples in the NiFi source code.

Building NiFi

Follow the Apache NiFi Development Quickstart to build NiFi from source. The steps in this post are based off version 0.0.2-incubating-SNAPSHOT, so make sure you have the latest code.

After building NiFi you can find the distribution at nifi/nifi-assembly/target:

ls -l nifi/nifi-assembly/target/
 archive-tmp
 maven-shared-archive-resources
 nifi-0.0.2-incubating-SNAPSHOT-bin
 nifi-0.0.2-incubating-SNAPSHOT-bin.tar.gz
 nifi-0.0.2-incubating-SNAPSHOT-bin.zip

You can run NiFi directory from nifi-0.0.2-incubating-SNAPSHOT-bin, or you can extract the tar to an external location. The advantage to extracting it somewhere else is that you won’t lose all of your configuration when you build NiFi.

Whichever you choose, we will call this directory NIFI_HOME from this point forward.

NOTE: This post was written at a type when there was no released version of the archetype, so it was a requirement to first build NiFi from source. In the future a released version of the archetype could be used without this requirement.

Creating a Processor Bundle

The easiest way to get started writing a custom processor is to use the Maven archetype included with NiFI. From a directory where you want to create your processor bundle, execute the following:

mvn archetype:generate -DarchetypeGroupId=org.apache.nifi -DarchetypeArtifactId=nifi-processor-bundle-archetype -DarchetypeVersion=0.0.2-incubating-SNAPSHOT -DnifiVersion=0.0.2-incubating-SNAPSHOT

This will prompt you to enter some information:

  • groupId - The Maven groupId of your new bundle
  • artifactId - The Maven artifactId of your new bundle, generally nifi-basename-bundle, where basename is specific to your bundle
  • version - The Maven version of your new bundle
  • artifactBaseName - Describes your bundle, should line up with the artifactId above
  • package - The Java package for your processors

For this example enter the following, accepting the default values for version and package:

Define value for property 'groupId': : org.apache.nifi
Define value for property 'artifactId': : nifi-helloworld-bundle
Define value for property 'version':  1.0-SNAPSHOT: :
Define value for property 'artifactBaseName': : helloworld
Define value for property 'package':  org.apache.nifi.processors.helloworld: :

This will generate a new Maven project called nifi-helloworld-bundle, with two sub-projects:

  • nifi-helloworld-processors - Contains processor implementations
  • nifi-helloworld-nar - Packages processors into a nar for deployment

The overall project structure looks like this:

nifi-helloworld-bundle
├── nifi-helloworld-nar
│   └── pom.xml
├── nifi-helloworld-processors
│   ├── pom.xml
│   └── src
│       ├── main
│       │   ├── java
│       │   │   └── org
│       │   │       └── apache
│       │   │           └── nifi
│       │   │               └── processors
│       │   │                   └── helloworld
│       │   │                       └── MyProcessor.java
│       │   └── resources
│       │       ├── META-INF
│       │       │   └── services
│       │       │       └── org.apache.nifi.processor.Processor
│       │       └── docs
│       │           └── org.apache.nifi.processors.helloworld.MyProcessor
│       │               └── index.html
│       └── test
│           └── java
│               └── org
│                   └── apache
│                       └── nifi
│                           └── processors
│                               └── helloworld
│                                   └── MyProcessorTest.java
└── pom.xml

The default processor generated by the archetype is called MyProcessor. If you decide to rename this processor, make sure to also update the org.apache.nifi.processor.Processor file. This file is used by NiFi to locate available processors, and the fully qualified class name of the processor must be listed here in order for it to be available from the user interface.

Deploying a Processor Bundle

Deploying a processor bundle consists of copying the nar to NIFI_HOME/lib, and restarting NiFi.

  1. Build the nifi-helloworld-bundle:

     cd nifi-helloworld-bundle
 mvn clean install

  2. Copy the nar to NIFI_HOME/lib:

     cp nifi-helloworld-nar/target/nifi-helloworld-nar-1.0-SNAPSHOT.nar NIFI_HOME/lib/

  3. Start NiFi:

     NIFI_HOME/bin/nifi.sh start

  4. Open a browser and navigate to http://localhost:8080/nifi

  5. Drag a new processor on to the flow using the Processor icon in the top-left

  6. Start typing MyProcessor in the Filter box

  7. Select MyProcessor and add it to the flow

Conclusion

At this point you should have a working processor bundle that you can easily deploy, and you can focus on implementing the processor!

If you ever change the Maven version or artifactId of your processor bundle, make sure to remove the old nars from NIFI_HOME/lib, and possibly remove NIFI_HOME/work/nar to clear old versions.

blog comments powered by Disqus