Packages

class KafkaAvroSourceFactory[K, V] extends SourceFactory with KafkaUniversalComponentTransformer[Source]

Base implementation of KafkaSource factory with Avro schema support. It is based on GenericNodeTransformation to - allow key and value type identification based on Schema Registry and - allow Context initialization with event's value, key and metadata You can provide schemas for both key and value. When useStringForKey = true (see KafkaConfig) the contents of event's key are treated as String (this is default scenario). Reader schema used in runtime is determined by topic and version. Reader schema can be different than schema used by writer (e.g. when writer produces event with new schema), in that case "schema evolution" may be required. For SpecificRecord use SpecificRecordKafkaAvroSourceFactory. Assumptions: 1. Every event that comes in has its key and value schemas registered in Schema Registry. 2. Avro payload must include schema id for both Generic and Specific records (to provide "schema evolution" we need to know the exact writers schema).

K

- type of event's key, used to determine if key object is Specific or Generic (for GenericRecords use Any)

V

- type of event's value, used to determine if value object is Specific or Generic (for GenericRecords use Any)

Linear Supertypes
KafkaUniversalComponentTransformer[Source], WithCachedTopicsExistenceValidator, TopicsExistenceValidator, SingleInputGenericNodeTransformation[Source], GenericNodeTransformation[Source], SourceFactory, Component, Serializable, Serializable, AnyRef, Any
Known Subclasses
SpecificRecordKafkaAvroSourceFactory, DelayedKafkaAvroSourceFactory
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaAvroSourceFactory
  2. KafkaUniversalComponentTransformer
  3. WithCachedTopicsExistenceValidator
  4. TopicsExistenceValidator
  5. SingleInputGenericNodeTransformation
  6. GenericNodeTransformation
  7. SourceFactory
  8. Component
  9. Serializable
  10. Serializable
  11. AnyRef
  12. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new KafkaAvroSourceFactory(schemaRegistryClientFactory: SchemaRegistryClientFactory, schemaBasedMessagesSerdeProvider: SchemaBasedSerdeProvider, processObjectDependencies: ProcessObjectDependencies, implProvider: KafkaSourceImplFactory[K, V])(implicit arg0: ClassTag[K], arg1: ClassTag[V])

Type Members

  1. type DefinedParameter = DefinedSingleParameter
    Definition Classes
    SingleInputGenericNodeTransformation → GenericNodeTransformation
  2. case class FinalResults extends TransformationStepResult with Product with Serializable
    Definition Classes
    GenericNodeTransformation
  3. type InputContext = ValidationContext
    Definition Classes
    SingleInputGenericNodeTransformation → GenericNodeTransformation
  4. case class NextParameters extends TransformationStepResult with Product with Serializable
    Definition Classes
    GenericNodeTransformation
  5. type NodeTransformationDefinition = PartialFunction[TransformationStep, TransformationStepResult]
    Definition Classes
    GenericNodeTransformation
  6. type State = KafkaAvroSourceFactoryState[K, V]
    Definition Classes
    KafkaAvroSourceFactory → GenericNodeTransformation
  7. case class TransformationStep extends Product with Serializable
    Definition Classes
    GenericNodeTransformation
  8. sealed trait TransformationStepResult extends AnyRef
    Definition Classes
    GenericNodeTransformation
  9. type WithError[V] = WriterT[Id, List[ProcessCompilationError], V]
    Definition Classes
    KafkaUniversalComponentTransformer

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  6. def contextTransformation(context: ValidationContext, dependencies: List[NodeDependencyValue])(implicit nodeId: NodeId): NodeTransformationDefinition
    Definition Classes
    KafkaAvroSourceFactory → GenericNodeTransformation
  7. def determineSchemaAndType(schemaDeterminer: AvroSchemaDeterminer, paramName: Option[String])(implicit nodeId: NodeId): Validated[ProcessCompilationError, (Option[RuntimeSchemaData[AvroSchema]], TypingResult)]
    Attributes
    protected
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def extractPreparedTopic(params: Map[String, Any]): PreparedKafkaTopic
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  11. def extractVersionOption(params: Map[String, Any]): SchemaVersionOption
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  12. def fallbackFinalResult(step: TransformationStep, inputContext: InputContext, outputVariable: Option[String])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
    Definition Classes
    GenericNodeTransformation
  13. val fallbackVersionOptionParam: Parameter
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  15. def getTopicParam(implicit nodeId: NodeId): WithError[Parameter]
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  16. def getVersionParam(versions: List[Integer]): Parameter
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  17. def getVersionParam(preparedTopic: PreparedKafkaTopic)(implicit nodeId: NodeId): WithError[Parameter]
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  18. def handleExceptionDuringTransformation(step: TransformationStep, inputContext: InputContext, outputVariable: Option[String], ex: Throwable)(implicit nodeId: NodeId): FinalResults
    Definition Classes
    GenericNodeTransformation
  19. def handleUnmatchedTransformationStep(step: TransformationStep, inputContext: InputContext, outputVariable: Option[String])(implicit nodeId: NodeId): FinalResults
    Definition Classes
    GenericNodeTransformation
  20. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  21. val implProvider: KafkaSourceImplFactory[K, V]
    Attributes
    protected
  22. def implementation(params: Map[String, Any], dependencies: List[NodeDependencyValue], finalState: Option[State]): Source
    Definition Classes
    KafkaAvroSourceFactory → GenericNodeTransformation
  23. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  24. val kafkaConfig: KafkaConfig
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer → WithCachedTopicsExistenceValidator
  25. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  26. def nextSteps(context: ValidationContext, dependencies: List[NodeDependencyValue])(implicit nodeId: NodeId): NodeTransformationDefinition
    Attributes
    protected
  27. def nodeDependencies: List[NodeDependency]
    Definition Classes
    KafkaAvroSourceFactory → GenericNodeTransformation
  28. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  29. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  30. val nullTopicOption: FixedExpressionValue
    Definition Classes
    KafkaUniversalComponentTransformer
  31. def paramsDeterminedAfterSchema: List[Parameter]
    Definition Classes
    KafkaAvroSourceFactory → KafkaUniversalComponentTransformer
  32. def parseVersionOption(versionOptionName: String): SchemaVersionOption
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  33. def prepareContextInitializer(dependencies: List[NodeDependencyValue], parameters: List[(String, DefinedParameter)], keyTypingResult: TypingResult, valueTypingResult: TypingResult): ContextInitializer[ConsumerRecord[K, V]]
    Attributes
    protected
  34. final def prepareFinalResultWithOptionalVariable(inputContext: InputContext, outputVariable: Option[(String, TypingResult)], state: Option[State])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
    Definition Classes
    GenericNodeTransformation
  35. def prepareKafkaConfig: KafkaConfig
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  36. def prepareKeySchemaDeterminer(preparedTopic: PreparedKafkaTopic): AvroSchemaDeterminer
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  37. def prepareSourceFinalErrors(context: ValidationContext, dependencies: List[NodeDependencyValue], parameters: List[(String, DefinedParameter)], errors: List[ProcessCompilationError])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
  38. def prepareSourceFinalResults(preparedTopic: PreparedKafkaTopic, valueValidationResult: Validated[ProcessCompilationError, (Option[RuntimeSchemaData[AvroSchema]], TypingResult)], context: ValidationContext, dependencies: List[NodeDependencyValue], parameters: List[(String, DefinedParameter)], errors: List[ProcessCompilationError])(implicit nodeId: NodeId): FinalResults
    Attributes
    protected
  39. def prepareTopic(topic: String): PreparedKafkaTopic
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  40. def prepareUniversalKeySchemaDeterminer(preparedTopic: PreparedKafkaTopic): ParsedSchemaDeterminer
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  41. def prepareUniversalValueSchemaDeterminer(preparedTopic: PreparedKafkaTopic, version: SchemaVersionOption): ParsedSchemaDeterminer
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  42. def prepareValueSchemaDeterminer(preparedTopic: PreparedKafkaTopic, version: SchemaVersionOption): AvroSchemaDeterminer
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  43. val processObjectDependencies: ProcessObjectDependencies
    Definition Classes
    KafkaAvroSourceFactory → KafkaUniversalComponentTransformer
  44. val schemaBasedMessagesSerdeProvider: SchemaBasedSerdeProvider
  45. def schemaParamStep(implicit nodeId: NodeId): NodeTransformationDefinition
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  46. lazy val schemaRegistryClient: SchemaRegistryClient
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
    Annotations
    @transient()
  47. val schemaRegistryClientFactory: SchemaRegistryClientFactory
    Definition Classes
    KafkaAvroSourceFactory → KafkaUniversalComponentTransformer
  48. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  49. def toString(): String
    Definition Classes
    AnyRef → Any
  50. val topicParamName: String
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  51. def topicParamStep(implicit nodeId: NodeId): NodeTransformationDefinition
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  52. def topicSelectionStrategy: TopicSelectionStrategy
    Attributes
    protected
    Definition Classes
    KafkaUniversalComponentTransformer
  53. final def validateTopic(topic: String): Validated[TopicExistenceValidationException, String]
    Definition Classes
    TopicsExistenceValidator
  54. final def validateTopics(topics: List[String]): Validated[TopicExistenceValidationException, List[String]]
    Definition Classes
    WithCachedTopicsExistenceValidator → TopicsExistenceValidator
  55. lazy val validator: CachedTopicsExistenceValidator
    Attributes
    protected
    Definition Classes
    WithCachedTopicsExistenceValidator
  56. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  57. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  58. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  59. object FinalResults extends Serializable
    Definition Classes
    GenericNodeTransformation

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated @deprecated
    Deprecated

    (Since version ) see corresponding Javadoc for more information.

Inherited from KafkaUniversalComponentTransformer[Source]

Inherited from WithCachedTopicsExistenceValidator

Inherited from TopicsExistenceValidator

Inherited from SingleInputGenericNodeTransformation[Source]

Inherited from GenericNodeTransformation[Source]

Inherited from SourceFactory

Inherited from Component

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped