Menon

தமிழ் |APACHE KAFKA WHAT ARE THE IMPORTANT CONSUMER CONFIGURATIONS EXPLAIN WITH EXAMPLE|InterviewDOT

LEARN APACHE KAFKA IN 2 MINUTES TAMIL TUTORIAL SERIES

Click here - https://www.youtube.com/channel/UCd0U_xlQxdZynq09knDszXA?sub_confirmation=1 to get notifications. Configuring Apache Kafka consumers is crucial for efficient and reliable message consumption. Here's an overview of important consumer configurations: 1. **bootstrap.servers**: List of host/port pairs to use for establishing initial connection to Kafka cluster. 2. **group.id**: ID of the consumer group the consumer belongs to. Required for group management and offset commits. 3. **auto.offset.reset**: Action to take when no initial offset is available or when an offset is out of range. Options include 'earliest' (start from beginning) or 'latest' (start from end). 4. **enable.auto.commit**: If true, consumer's offsets are automatically committed to Kafka at regular intervals defined by auto.commit.interval.ms. 5. **auto.commit.interval.ms**: Frequency (in milliseconds) at which offsets are auto-committed if enable.auto.commit is true. 6. **max.poll.records**: Maximum number of records returned in each poll request to Kafka broker. 7. **max.poll.interval.ms**: Maximum delay (in milliseconds) between successive calls to poll() method. If exceeded, consumer is considered dead and its partitions are reassigned. 8. **fetch.min.bytes**: Minimum amount of data (in bytes) that broker should return for a fetch request. Broker waits until it has at least this much data to send to consumer. 9. **fetch.max.wait.ms**: Maximum amount of time (in milliseconds) broker waits to receive fetch.min.bytes of data before responding to consumer. 10. **fetch.max.bytes**: Maximum amount of data (in bytes) that consumer will attempt to fetch in a single request to broker. 11. **session.timeout.ms**: Duration (in milliseconds) after which consumer is considered dead if it hasn't sent any heartbeats. 12. **heartbeat.interval.ms**: Frequency (in milliseconds) at which consumer sends heartbeat to indicate its liveness to the group coordinator. 13. **max.partition.fetch.bytes**: Maximum amount of data (in bytes) fetched from a single partition at a time. 14. **request.timeout.ms**: Maximum amount of time (in milliseconds) consumer waits for a response from broker before considering it as a failure. 15. **isolation.level**: Guarantees that consumer reads only 'read_committed' or 'read_uncommitted' data. Affects transactional and non-transactional messages. 16. **partition.assignment.strategy**: Strategy used for assigning partitions to consumers within a group. Default is 'range'. 17. **max.poll.interval.ms**: Maximum time (in milliseconds) that can elapse between two consecutive poll invocations before consumer is considered inactive and rebalancing is triggered. 18. **exclude.internal.topics**: Whether to exclude internal topics (such as __consumer_offsets) from subscription. 19. **fetch.max.partition.bytes**: Maximum amount of data (in bytes) returned by broker for each partition in fetch response. 20. **check.crcs**: Whether to check message CRC32 during deserialization. Adds overhead but ensures data integrity. 21. **receive.buffer.bytes**: Size (in bytes) of the TCP receive buffer to use when reading data from sockets. 22. **max.partition.fetch.bytes**: Maximum amount of data (in bytes) fetched from each partition at a time. 23. **client.id**: ID string to pass to server when making requests. Useful for tracing requests. 24. **max.in.flight.requests.per.connection**: Maximum number of unacknowledged requests broker can handle before blocking further requests. 25. **reconnect.backoff.ms**: Time (in milliseconds) to wait before attempting to reconnect to broker after connection failure. 26. **retry.backoff.ms**: Time (in milliseconds) to wait before retrying failed request to broker. 27. **security.protocol**: Protocol used to communicate with brokers. Options include 'PLAINTEXT', 'SSL', 'SASL_PLAINTEXT', 'SASL_SSL', etc. 28. **ssl.truststore.location**: Location of trust store file used to verify server's certificate. 29. **ssl.truststore.password**: Password for trust store file. 30. **ssl.keystore.location**: Location of key store file used for client authentication. 31. **ssl.keystore.password**: Password for key store file. 32. **ssl.key.password**: Password for private key in key store. 33. **ssl.enabled.protocols**: List of SSL protocols enabled for communication with brokers. 34. **ssl.keystore.type**: Type of key store file (e.g., JKS, PKCS12). 35. **ssl.truststore.type**: Type of trust store file (e.g., JKS, PKCS12). 36. **ssl.endpoint.identification.algorithm**: Algorithm used to determine server's identity during SSL handshake. 37. **sasl.mechanism**: SASL mechanism used for authentication (e.g., PLAIN, GSSAPI). 38. **sasl.kerberos.service.name**: Name of the Kerberos service principal. 39. **sasl.kerberos.kinit.cmd**: Command to obtain Kerberos ticket. 40. **sasl.kerberos.min.time.before.relogin**: Minimum time (in milliseconds) before re-authenticating using Kerberos.

About the course

**Apache Kafka Messaging System in 4000 Characters:** **Introduction:** Apache Kafka is an open-source distributed streaming platform designed for building real-time data pipelines and streaming applications. Developed by the Apache Software Foundation, Kafka has become a cornerstone technology for organizations dealing with large-scale, real-time data processing. **Key Concepts:** 1. **Publish-Subscribe Model:** - Kafka follows a publish-subscribe model where producers publish messages to topics, and consumers subscribe to those topics to receive the messages. This decouples data producers and consumers, enabling scalable and flexible architectures. 2. **Topics and Partitions:** - Data is organized into topics, acting as logical channels for communication. Topics are divided into partitions, allowing parallel processing and scalability. Each partition is a linear, ordered sequence of messages. 3. **Brokers and Clusters:** - Kafka brokers form a cluster, ensuring fault tolerance and high availability. Brokers manage the storage and transmission of messages. Kafka clusters can scale horizontally by adding more brokers, enhancing both storage and processing capabilities. 4. **Producers and Consumers:** - Producers generate and send messages to Kafka topics, while consumers subscribe to topics and process the messages. This separation allows for the decoupling of data producers and consumers, supporting scalability and flexibility. 5. **Event Log:** - Kafka maintains an immutable, distributed log of records (messages). This log serves as a durable event store, allowing for the replay and reprocessing of events. Each message in the log has a unique offset. 6. **Scalability:** - Kafka's scalability is achieved through partitioning and distributed processing. Topics can be partitioned, and partitions can be distributed across multiple brokers, enabling horizontal scaling to handle large volumes of data. **Use Cases:** 1. **Real-time Data Streams:** - Kafka excels in handling and processing real-time data streams, making it suitable for use cases like monitoring, fraud detection, and analytics where timely insights are crucial. 2. **Log Aggregation:** - It serves as a powerful solution for aggregating and centralizing logs from various applications and services. Kafka's durability ensures that logs are reliably stored for analysis and troubleshooting. 3. **Messaging Backbone:** - Kafka acts as a robust and fault-tolerant messaging system, connecting different components of a distributed application. Its durability and reliability make it a reliable backbone for messaging. 4. **Event Sourcing:** - Kafka is often used in event sourcing architectures where changes to application state are captured as a sequence of events. This approach enables reconstruction of the application state at any point in time. 5. **Microservices Integration:** - Kafka facilitates communication between microservices in a distributed system. It provides a resilient and scalable mechanism for asynchronous communication, ensuring loose coupling between services. **Components:** 1. **ZooKeeper:** - Kafka relies on Apache ZooKeeper for distributed coordination, managing configuration, and electing leaders within the Kafka cluster. ZooKeeper ensures the stability and coordination of Kafka brokers. 2. **Producer API:** - Producers use Kafka's Producer API to publish messages to topics. The API supports asynchronous and synchronous message publishing, providing flexibility for different use cases. 3. **Consumer API:** - Consumers use Kafka's Consumer API to subscribe to topics and process messages. Consumer groups allow parallel processing and load balancing, ensuring efficient utilization of resources. 4. **Connect API:** - Kafka Connect enables the integration of Kafka with external systems. Connectors, available for various data sources and sinks, simplify the development of data pipelines between Kafka and other systems. 5. **Streams API:** - Kafka Streams API facilitates the development of stream processing applications directly within Kafka. It enables transformations and analytics on streaming data, supporting real-time processing scenarios. **Reliability and Durability:** 1. **Replication:** - Kafka ensures data durability through replication. Each partition has a leader and multiple followers, with data replicated across brokers. This replication mechanism ensures fault tolerance and data redundancy. 2. **Retention Policies:** - Kafka allows the configuration of retention policies for topics. This determines how long messages are retained in a topic. Retention policies support both real-time and historical data analysis. **Ecosystem and Integration:**