<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Kafka Connect on Florian Hussonnois — Apache Kafka, Kestra &amp; Data Streaming</title><link>https://www.fhussonnois.dev/tags/kafka-connect/</link><description>Recent content in Kafka Connect on Florian Hussonnois — Apache Kafka, Kestra &amp; Data Streaming</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Wed, 18 Oct 2023 00:00:00 +0000</lastBuildDate><atom:link href="https://www.fhussonnois.dev/tags/kafka-connect/index.xml" rel="self" type="application/rss+xml"/><item><title>Kafka Connect + Jikkou: Easily manage Kafka connectors</title><link>https://www.fhussonnois.dev/posts/kafka-connect-jikkou-easily-manage-kafka-connectors/</link><pubDate>Wed, 18 Oct 2023 00:00:00 +0000</pubDate><guid>https://www.fhussonnois.dev/posts/kafka-connect-jikkou-easily-manage-kafka-connectors/</guid><description>&lt;p&gt;&lt;img src="https://www.fhussonnois.dev/images/posts/kafka-connect-jikkou/cover.jpg" alt="Photo by Miltiadis Fragkidis on Unsplash" loading="lazy"&gt;
&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Photo by &lt;a href="https://unsplash.com/fr/@_miltiadis_" target="_blank" rel="noopener noreferrer"&gt;Miltiadis Fragkidis&lt;/a&gt;
 on &lt;a href="https://unsplash.com" target="_blank" rel="noopener noreferrer"&gt;Unsplash&lt;/a&gt;
&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;This blog post was originally published on &lt;a href="https://medium.com/@fhussonnois/kafka-connect-jikkou-easily-manage-kafka-connectors-d4dd2b8a560c" target="_blank" rel="noopener noreferrer"&gt;Medium&lt;/a&gt;
&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Kafka Connect is a widely used solution to stream data into &lt;a href="https://kafka.apache.org/documentation/" target="_blank" rel="noopener noreferrer"&gt;Apache Kafka&lt;/a&gt;
 and to transfer data from Kafka to external systems for further use, such as online analytical processing.&lt;/p&gt;
&lt;p&gt;One of the main advantages of Kafka Connect is its ease of use. Basically, all you have to do is define the JSON configuration corresponding to the connector you want to run and push it to the REST API of your Kafka Connect cluster to start data integration.&lt;/p&gt;</description></item><item><title>How to build a real-time analytics platform using Kafka, ksqlDB and ClickHouse ?</title><link>https://www.fhussonnois.dev/posts/how-to-build-a-real-time-analytical-platform-using-kafka-ksqldb-and-clickhouse/</link><pubDate>Tue, 02 Jun 2020 00:00:00 +0000</pubDate><guid>https://www.fhussonnois.dev/posts/how-to-build-a-real-time-analytical-platform-using-kafka-ksqldb-and-clickhouse/</guid><description>&lt;p&gt;&lt;img src="https://www.fhussonnois.dev/images/posts/kafka-ksqldb-clickhouse/cover.png" alt="Apache Kafka &amp;#43; ksqlDB &amp;#43; ClickHouse &amp;#43; Superset = Blazing Fast Analytics Platform" loading="lazy"&gt;
&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Apache Kafka + ksqlDB + ClickHouse + Superset = Blazing Fast Analytics Platform&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;This blog post was originally published on &lt;a href="https://medium.com/streamthoughts/how-to-build-a-real-time-analytical-platform-using-kafka-ksqldb-and-clickhouse-bfabd65d05e4" target="_blank" rel="noopener noreferrer"&gt;Medium&lt;/a&gt;
&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Recently at StreamThoughts, we have looked at different open-source OLAP databases that we could quickly experiment in a streaming architecture, based on the &lt;a href="https://kafka.apache.org/" target="_blank" rel="noopener noreferrer"&gt;Apache Kafka&lt;/a&gt;
 platform.&lt;/p&gt;
&lt;p&gt;Our goal was to be able to respond to analytical needs on large volumes of data that were ingested in real-time. For this, we were looking for a solution that would allow us to execute ad-hoc queries, interactively, with “acceptable” latencies (a few seconds or more).&lt;/p&gt;</description></item><item><title>Kafka Connect FilePulse - One Connector to Ingest them All!</title><link>https://www.fhussonnois.dev/posts/kafka-connect-filepulse-one-connector-to-ingest-them-all/</link><pubDate>Fri, 24 Jan 2020 00:00:00 +0000</pubDate><guid>https://www.fhussonnois.dev/posts/kafka-connect-filepulse-one-connector-to-ingest-them-all/</guid><description>&lt;p&gt;&lt;img src="https://www.fhussonnois.dev/images/posts/kafka-connect-filepulse/cover.jpg" alt="Photo by Brandon Green on Unsplash" loading="lazy"&gt;
&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Photo by Brandon Green on &lt;a href="https://unsplash.com" target="_blank" rel="noopener noreferrer"&gt;Unsplash&lt;/a&gt;
&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;This blog post was originally published on &lt;a href="https://medium.com/streamthoughts/kafka-connect-filepulse-one-connector-to-ingest-them-all-faed018a725c" target="_blank" rel="noopener noreferrer"&gt;Medium&lt;/a&gt;
&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Most of the projects I’ve worked on in the last few years have involved ingesting data into systems such as Apache Kafka® and Apache Hadoop® to perform processing and data enrichment both in real-time and in batch.&lt;/p&gt;
&lt;p&gt;One of the recurring challenges of each of these projects has always been to manage the complexity of legacy systems in which data was frequently exported, shared and integrated through the use of files.&lt;/p&gt;</description></item><item><title>Apache Kafka Rebalance Protocol, or the magic behind your streams applications</title><link>https://www.fhussonnois.dev/posts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications/</link><pubDate>Tue, 05 Nov 2019 00:00:00 +0000</pubDate><guid>https://www.fhussonnois.dev/posts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications/</guid><description>&lt;p&gt;&lt;em&gt;This blog post was originally published on &lt;a href="https://medium.com/streamthoughts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications-e94baf68e4f2" target="_blank" rel="noopener noreferrer"&gt;Medium&lt;/a&gt;
&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Since Apache Kafka 2.3.0, the internal Rebalance Protocol, which is especially used by Kafka Connect and consumers, has undergone several major changes.&lt;/p&gt;
&lt;p&gt;The Rebalance Protocol is not something simple and can sometimes look like magic. In this post, I propose to go back to the foundation of this protocol, which is at the heart of the Apache Kafka consumption mechanism. Then, we will discuss its limitations and current improvements.&lt;/p&gt;</description></item></channel></rss>