Custom Processors for Apache NiFi
This post will cover the initial steps to get started developing a custom processor for Apache NiFi. The focus will be on setting up a new processor bundle and and deploying it.
The specifics of implementing a processor can be found in the NiFi Developer’s Guide, as well as many examples in the NiFi source code.
Follow the Apache NiFi Development Quickstart to build NiFi from source. The steps in this post are based off version 0.0.2-incubating-SNAPSHOT, so make sure you have the latest code.
After building NiFi you can find the distribution at nifi/nifi-assembly/target:
ls -l nifi/nifi-assembly/target/ archive-tmp maven-shared-archive-resources nifi-0.0.2-incubating-SNAPSHOT-bin nifi-0.0.2-incubating-SNAPSHOT-bin.tar.gz nifi-0.0.2-incubating-SNAPSHOT-bin.zip
You can run NiFi directory from nifi-0.0.2-incubating-SNAPSHOT-bin, or you can extract the tar to an external location. The advantage to extracting it somewhere else is that you won’t lose all of your configuration when you build NiFi.
Whichever you choose, we will call this directory NIFI_HOME from this point forward.
NOTE: This post was written at a type when there was no released version of the archetype, so it was a requirement to first build NiFi from source. In the future a released version of the archetype could be used without this requirement.
Creating a Processor Bundle
The easiest way to get started writing a custom processor is to use the Maven archetype included with NiFI. From a directory where you want to create your processor bundle, execute the following:
mvn archetype:generate -DarchetypeGroupId=org.apache.nifi -DarchetypeArtifactId=nifi-processor-bundle-archetype -DarchetypeVersion=0.0.2-incubating-SNAPSHOT -DnifiVersion=0.0.2-incubating-SNAPSHOT
This will prompt you to enter some information:
- groupId - The Maven groupId of your new bundle
- artifactId - The Maven artifactId of your new bundle, generally nifi-basename-bundle, where basename is specific to your bundle
- version - The Maven version of your new bundle
- artifactBaseName - Describes your bundle, should line up with the artifactId above
- package - The Java package for your processors
For this example enter the following, accepting the default values for version and package:
Define value for property 'groupId': : org.apache.nifi Define value for property 'artifactId': : nifi-helloworld-bundle Define value for property 'version': 1.0-SNAPSHOT: : Define value for property 'artifactBaseName': : helloworld Define value for property 'package': org.apache.nifi.processors.helloworld: :
This will generate a new Maven project called nifi-helloworld-bundle, with two sub-projects:
- nifi-helloworld-processors - Contains processor implementations
- nifi-helloworld-nar - Packages processors into a nar for deployment
The overall project structure looks like this:
nifi-helloworld-bundle ├── nifi-helloworld-nar │ └── pom.xml ├── nifi-helloworld-processors │ ├── pom.xml │ └── src │ ├── main │ │ ├── java │ │ │ └── org │ │ │ └── apache │ │ │ └── nifi │ │ │ └── processors │ │ │ └── helloworld │ │ │ └── MyProcessor.java │ │ └── resources │ │ ├── META-INF │ │ │ └── services │ │ │ └── org.apache.nifi.processor.Processor │ │ └── docs │ │ └── org.apache.nifi.processors.helloworld.MyProcessor │ │ └── index.html │ └── test │ └── java │ └── org │ └── apache │ └── nifi │ └── processors │ └── helloworld │ └── MyProcessorTest.java └── pom.xml
The default processor generated by the archetype is called MyProcessor. If you decide to rename this processor, make sure to also update the org.apache.nifi.processor.Processor file. This file is used by NiFi to locate available processors, and the fully qualified class name of the processor must be listed here in order for it to be available from the user interface.
Deploying a Processor Bundle
Deploying a processor bundle consists of copying the nar to NIFI_HOME/lib, and restarting NiFi.
Build the nifi-helloworld-bundle:
cd nifi-helloworld-bundle mvn clean install
Copy the nar to NIFI_HOME/lib:
cp nifi-helloworld-nar/target/nifi-helloworld-nar-1.0-SNAPSHOT.nar NIFI_HOME/lib/
Open a browser and navigate to http://localhost:8080/nifi
Drag a new processor on to the flow using the Processor icon in the top-left
Start typing MyProcessor in the Filter box
Select MyProcessor and add it to the flow
At this point you should have a working processor bundle that you can easily deploy, and you can focus on implementing the processor!
If you ever change the Maven version or artifactId of your processor bundle, make sure to remove the old nars from NIFI_HOME/lib, and possibly remove NIFI_HOME/work/nar to clear old versions.
blog comments powered by Disqus