Odtnhj

I am trying to setup a data pipeline using Kafka.
Data go in (with producers), get processed, enriched and cleaned and move out to different databases or storage (with consumers or Kafka connect).

But where do you run the actual pipeline processing code to enrich and clean the data? Should it be part of the producers or the consumers? I think I missed something.

In the use case of a data pipeline the Kafka clients could serve both as a consumer and producer.

For example, if you have raw data being streamed into ClientA where it is being cleaned before being passed to ClientB for enrichment then ClientA is serving as a consumer (listening to a topic for raw data) and a producer (publishing cleaned data to a topic).

Where you draw those boundaries is a separate question.

It can be part of either producer or consumer.

Or you could setup an environment dedicated to something like Kafka Streams processes or a KSQL cluster

It is possible either ways.Consider all possible options , choose an option which suits you best. Lets assume you have a source, raw data in csv or some DB(Oracle) and you want to do your ETL stuff and load it back to some different datastores

1) Use kafka connect to produce your data to kafka topics.
 Have a consumer which would consume off of these topics(could Kstreams, Ksql or Akka, Spark).
 Produce back to a kafka topic for further use or some datastore, any sink basically

This has the benefit of ingesting your data with little or no code using kafka connect as it is easy to set up kafka connect source producers.

2) Write custom producers, do your transformations in producers before 
writing to kafka topic or directly to a sink unless you want to reuse this produced data 
for some further processing.
Read from kafka topic and do some further processing and write it back to persistent store. 

It all boils down to your design choice, the thoughput you need from the system, how complicated your data structure is.