Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Apache NiFi SDLC Improvements

1 594 vues

Publié le

Overview of SDLC improvements with Apache NiFi 1.10.0 and Apache NiFi Registry 0.5.0.

Publié dans : Logiciels
  • Identifiez-vous pour voir les commentaires

Apache NiFi SDLC Improvements

  1. 1. © Cloudera, Inc. All rights reserved. Apache NiFi SDLC Improvements Bryan Bende / @bbende November 2019
  2. 2. © Cloudera, Inc. All rights reserved. 2© Cloudera, Inc. All rights reserved. OUTLINE • NiFi 1.10.0 • Parameterized Flows • Force commit • Auto-select external controller services • Track enabled/disabled state • Change version with nested versioning • NiFi Registry 0.5.0 • Granular proxy permissions • Public buckets • Versioned Extension Bundles
  3. 3. © Cloudera, Inc. All rights reserved. 3© Cloudera, Inc. All rights reserved. PARAMETERIZED FLOWS
  4. 4. © Cloudera, Inc. All rights reserved. 4© Cloudera, Inc. All rights reserved. PROBLEMS • Variables are referenced through expression language (EL)… • Some properties don’t support EL and can’t be parameterized • Can’t apply access control because references are ambiguous • Ex: ${foo} could be a flow file attribute, variable, system property, or environment variable • Without access control, can’t have sensitive variables • Without sensitive variables, can’t parameterize sensitive properties!
  5. 5. © Cloudera, Inc. All rights reserved. 5© Cloudera, Inc. All rights reserved. SOLUTION – INTRODUCE PARAMETER CONTEXTS • Parameter contexts created outside of the flow • Context has a name, description, and one or more parameters • Parameter has a name, description, and sensitivity flag • Process group can be bound to one parameter context • Components in the process group can reference parameters in the bound context • New syntax for referencing parameters in properties: #{param-name} • All properties support parameters regardless of expression language • Sensitive properties can only reference sensitive parameters (vice versa) • Integration with NiFi registry when migrating flow between environments
  6. 6. © Cloudera, Inc. All rights reserved. 6© Cloudera, Inc. All rights reserved. MANAGE PARAMETER CONTEXTS • Control who can create parameter contexts • Control “view” & “modify” permissions for each context • Sensitive parameter values encrypted and never returned
  7. 7. © Cloudera, Inc. All rights reserved. 7© Cloudera, Inc. All rights reserved. BIND PROCESS GROUP TO CONTEXT • Configure process group to select a parameter context • Select from contexts the current user has “view” permissions for • Requires “modify” on process group
  8. 8. © Cloudera, Inc. All rights reserved. 8© Cloudera, Inc. All rights reserved. REFERENCE PARAMETERS IN FLOW • Reference parameters in any property, regardless of EL support • Sensitive properties can only reference sensitive parameters • Easily promote property values to parameters from up-arrow icon
  9. 9. © Cloudera, Inc. All rights reserved. 9© Cloudera, Inc. All rights reserved. VERSION CONTROL FLOW WITH PARAMETERS "parameterContexts" : { "SFTP Params" : { "name" : "SFTP Params", "parameters" : [ { "name" : "sftp.password", "sensitive" : true }, { "name" : "sftp.host", "sensitive" : false, "value" : "localhost" }, { "name" : "sftp.user", "sensitive" : false, "value" : "myuser" } ] } • Saved to registry with snapshots of referenced parameter contexts • Values of sensitive parameters scrubbed, set once after importing to target environment • Sensitive properties in versioned flow retain parameter references like #{password}
  10. 10. © Cloudera, Inc. All rights reserved. 10© Cloudera, Inc. All rights reserved. IMPORT/UPGRADE VERSION CONTROLLED FLOW • For each parameter context in incoming versioned flow… • If no existing context with same name, create new context using initial values from versioned flow • Requires permissions to create a new context • If existing context with same name, add new parameters not already in existing context • Requires “view” & "modify” permissions to the existing context • After import/upgrade, set sensitive parameter values in given contexts
  11. 11. © Cloudera, Inc. All rights reserved. 11© Cloudera, Inc. All rights reserved. MANAGE PARAMETERS WITH NIFI CLI • CLI commands for… • create-param-context • list-param-contexts • get-param-context • set-param • delete-param • pg-set-param-context • export-param-context • import-param-context • merge-param-context
  12. 12. © Cloudera, Inc. All rights reserved. 12© Cloudera, Inc. All rights reserved. GENERAL NIFI SDLC IMPROVEMENTS
  13. 13. © Cloudera, Inc. All rights reserved. 13© Cloudera, Inc. All rights reserved. PROBLEM – CAN’T PROCEED AFTER REVERTING • If latest version of a flow is bad, change version back to previous (i.e. revert), BUT now local changes put flow into conflict state • No way to move forward based on previous version
  14. 14. © Cloudera, Inc. All rights reserved. 14© Cloudera, Inc. All rights reserved. SOLUTION – FORCE COMMIT • Allow committing local changes as next version regardless of available upgrades (i.e. force commit next version)
  15. 15. © Cloudera, Inc. All rights reserved. 15© Cloudera, Inc. All rights reserved. PROBLEM – UNLINKED CONTROLLER SERVICES • If a component references a controller service from outside the versioned process group, service must be re-selected on import (first time only)
  16. 16. © Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved. SOLUTION – AUTO-SLECET CONTROLLER SERVICES BY NAME • Track names of external controller services referenced by versioned flow • During import, find all services from parent groups… • If only one service matching the desired type with same name, auto-select • If multiple services matching desired type with same name, require user to select • Example: • Dev – service named ‘DBCPConnectionPool’ in root group • Prod - service name ‘DBCPConnectionPool’ in root group • Import flow from dev environment to prod environment • Processors referencing ‘DBCPConnectionPool’ get correctly linked to prod service by name
  17. 17. © Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved. OTHER IMPROVEMENTS… • Store enabled/disabled state of components in registry • Retain appropriate state on import of versioned flow • https://issues.apache.org/jira/browse/NIFI-6025 • Recursively change version on nested versioned process groups when changing version on a parent • https://issues.apache.org/jira/browse/NIFI-6314 • Ignore changes in local flow caused by new properties with default values • https://issues.apache.org/jira/browse/NIFI-6028
  18. 18. © Cloudera, Inc. All rights reserved. 18© Cloudera, Inc. All rights reserved. NIFI REGISTRY IMPROVEMENTS
  19. 19. © Cloudera, Inc. All rights reserved. 19© Cloudera, Inc. All rights reserved. PROBLEM – PROD SHOULDN’T BE ABLE MODIFY REGISTRY • Many teams want to enforce a development workflow • Dev -> Staging -> Prod • If a problem is found in staging or prod, start back in dev • Previously no way to enforce that a NiFi instance can’t write to a registry
  20. 20. © Cloudera, Inc. All rights reserved. 20© Cloudera, Inc. All rights reserved. SOLUTION – GRANULAR PROXY PERMISSIONS • Proxy permissions allow NiFi to make requests to registry on behalf of an end user • Previously a single permission for Proxy (yes or no) • Proxy permissions now split into ‘Read’, ‘Write’, ‘Delete’ • A proxy with only ‘Read’ can import flows, but can’t save new versions
  21. 21. © Cloudera, Inc. All rights reserved. 21© Cloudera, Inc. All rights reserved. PROBLEM – ANONYMOUS ACCESS TO SOME BUCKETS • Secured registry requires all access to come from authenticated users • No way to make some items public so that anyone can retrieve them • Requires all users to have accounts
  22. 22. © Cloudera, Inc. All rights reserved. 22© Cloudera, Inc. All rights reserved. SOLUTION – DECLARE BUCKETS PUBLICLY VISIBLE • Allow a bucket to be marked as public • All items in a public bucket are read-only for unauthenticated users • Configure anonymous access • nifi.registry.security.needClientAuth=false • When no client cert is presented, user sent to home page seeing publicly visible items
  23. 23. © Cloudera, Inc. All rights reserved. 23© Cloudera, Inc. All rights reserved. PROBLEM – VERSION CONTROL OF EXTENSIONS • Versioned flows reference specific versions of extensions bundles { "type" : "org.apache.nifi.processors.standard.LookupRecord", "bundle" : { "artifact" : "nifi-standard-nar", "group" : "org.apache.nifi", "version" : "1.10.0" } ... } • In order to deploy a flow, we also need the correct extensions bundles • Previously no way to version control bundles along side the flows
  24. 24. © Cloudera, Inc. All rights reserved. 24© Cloudera, Inc. All rights reserved. SOLUTION – VERSIONED EXTENSION BUNDLES • New type of versioned item in registry – ‘bundle’ • Currently one type of bundle – ‘NAR’ • Bundle must provide extension manifest (more info later) • Registry REST API for interacting with bundles • Bundles show in registry UI similar to flows
  25. 25. © Cloudera, Inc. All rights reserved. 25© Cloudera, Inc. All rights reserved. VERSIONED BUNDLES - DEEPER DIVE
  26. 26. © Cloudera, Inc. All rights reserved. 26© Cloudera, Inc. All rights reserved. EXTENSION MANIFESTS • Extension manifest describes all extensions contained in the bundle • XSD • https://gist.github.com/bbende/8df60c186bd94ed1dbfd42d61cfc63ef • Example • https://github.com/apache/nifi-registry/blob/master/nifi-registry-core/nifi-registry-bundle- utils/src/test/resources/descriptors/extension-manifest-hadoop-nar.xml • Plan to support different types of bundles for NiFi, MiNiFi CPP, etc. • Same extension manifest regardless of bundle type • Extractors to read extension manifest from given bundle types
  27. 27. © Cloudera, Inc. All rights reserved. 27© Cloudera, Inc. All rights reserved. NAR BUNDLES • NAR Maven Plugin version 1.3.1 generates extension manifests • Requires NAR built against nifi-api 1.10.0 • Example from nifi-hadoop-nar META-INF/ ├── docs | ├── additional-details│ | | ├── org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile│ | | | └── additionalDetails.html│ | | ├── org.apache.nifi.processors.hadoop.ListHDFS│ | | | └── additionalDetails.html│ | | └── org.apache.nifi.processors.hadoop.PutHDFS│ | | | └── additionalDetails.html│ | └── extension-manifest.xml
  28. 28. © Cloudera, Inc. All rights reserved. 28© Cloudera, Inc. All rights reserved. REGISTRY REST API • Consult Swagger documentation at: • http://<registry-host>:18080/nifi-registry-api/swagger/ui.html • Consult Admin Guide at: • https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html#manage-bundles
  29. 29. © Cloudera, Inc. All rights reserved. 29© Cloudera, Inc. All rights reserved. NIFI CLI • Commands to make working with registry REST API easier… • upload-bundle • upload-bundles • download-bundle • list-bundle-groups • list-bundle-artifacts • list-bundle-versions • list-extensions • list-extension-tags
  30. 30. © Cloudera, Inc. All rights reserved. 30© Cloudera, Inc. All rights reserved. EXAMPLE – GENERATE AND BUILD NAR mvn archetype:generate -DarchetypeGroupId=org.apache.nifi -DarchetypeArtifactId=nifi-processor-bundle-archetype -DarchetypeVersion=1.10.0 -DnifiVersion=1.10.0 Define value for property 'groupId': org.apache.nifi Define value for property 'artifactId': nifi-test-bundle Define value for property 'version' 1.0-SNAPSHOT: : 1.0.0 Define value for property 'artifactBaseName': test Define value for property 'package' org.apache.nifi.processors.test: : cd nifi-test-bundle mvn clean package [1] https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
  31. 31. © Cloudera, Inc. All rights reserved. 31© Cloudera, Inc. All rights reserved. EXAMPLE – UPLOAD BUNDLE • Download nifi-toolkit-1.10.0-bin.tar.gz from https://nifi.apache.org/download.html • Launch CLI from nifi-toolkit-1.10.0/bin/cli.sh • Execute upload-bundle command: • registry upload-bundle -u http://localhost:18080 -b 1005e90f-5751-4f10-8ae5- 69e0961fc02f -ebf /path/to/nifi-test-nar-1.0.0.nar -ebt nifi-nar
  32. 32. © Cloudera, Inc. All rights reserved. 32© Cloudera, Inc. All rights reserved. EXAMPLE – VIEW IN REGISTRY UI • Navigate to the registry UI and view bundle as a versioned item
  33. 33. © Cloudera, Inc. All rights reserved. 33© Cloudera, Inc. All rights reserved. EXAMPLE - BROWSE EXTENSION REPOSITORY API • Registry REST API exposes a hierarchical linked API for browsing bundles • Level 1 – Buckets the user is authorized for • http://localhost:18080/nifi-registry-api/extension-repository • Level 2 – Bundle group ids within a selected bucket • http://localhost:18080/nifi-registry-api/extension-repository/Bundles • Level 3 – Bundle artifact ids within a selected group • http://localhost:18080/nifi-registry-api/extension-repository/Bundles/org.apache.nifi • Level 4 – Bundle versions within a selected artifact • http://localhost:18080/nifi-registry-api/extension-repository/Bundles/org.apache.nifi/nifi-test-nar • Level 5 – Version specific info (download, checksum, docs) • http://localhost:18080/nifi-registry-api/extension-repository/Bundles/org.apache.nifi/nifi-test-nar/1.0.0
  34. 34. © Cloudera, Inc. All rights reserved. 34© Cloudera, Inc. All rights reserved. EXAMPLE – DOWNLOAD BUNDLE • Use CLI to download bundle to NiFi’s auto-load directory… • registry download-bundle -u http://localhost:18080 -bn "Bundles" -gr org.apache.nifi -ar nifi-test-bundle -ver 1.0.0 -od /path/to/nifi-home/extensions • Alternatively, curl can be used: • curl http://localhost:18080/nifi-registry-api/extension- repository/Bundles/org.apache.nifi/nifi-test-nar/1.0.0/content > /path/to/nifi-home/nifi- test-nar-1.0.0.nar • NAR will automatically load after a few seconds • Currently requires hard refresh of NiFi UI to show in the ‘Add Processor’ list
  35. 35. © Cloudera, Inc. All rights reserved. THANK YOU

×