Confluent reimagines data pipelines with Stream Designer

Confluent, the data streaming pioneer, has announced Stream Designer, a visual interface that enables developers to build and deploy streaming data pipelines in minutes. 

This point-and-click visual builder is a significant advancement toward democratising data streams, making them accessible to developers beyond specialised Apache Kafka experts. 

With more teams able to rapidly build and iterate on streaming pipelines, organisations can quickly connect more data throughout their business for agile development and better, faster, in-the-moment decision-making.

"We are in the middle of a major technological shift, where data streaming is making real time the new normal, enabling new business models, better customer experiences, and more efficient operations," says Jay Kreps, Co-founder and CEO, Confluent. 

"With Stream Designer we want to democratise this movement towards data streaming and make real time the default for all data flow in an organisation."

Confluent is the data streaming platform pioneering a new category of data infrastructure that sets data in motion. 

Confluent's cloud-native offering is the foundational platform for data in motion. It is designed to be the intelligent connective tissue enabling real-time data from multiple sources to stream across the organisation constantly. 

With Confluent, organisations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations.

In the streaming era, data streaming is the default mode of data operations for successful modern businesses. The streaming technologies that were once at the edges have become core to critical business functions. This shift is fuelled by the growing demand to deliver data instantaneously and scalably across a full range of customer experiences and business operations. Traditional batch processing can no longer keep pace with increasing use cases that depend on sub-millisecond updates across an ever-expansive set of data sources.

Thus, organisations seek ways to accelerate their data streaming initiatives as more of their business is operating in real-time. 

Kafka is the de facto standard for data streaming, enabling over 80% of Fortune 100 companies to handle large volumes and varieties of data in real-time reliably. 

However, building streaming data pipelines on open-source Kafka requires large teams of highly specialised engineering talent and time-consuming development spread across multiple tools. This puts pervasive data streaming out of reach for many organisations and leaves legacy pipelines clogged with stale and outdated data.

"A rising number of organisations are realising streaming data is imperative to achieving innovation and maintaining a healthy business," says Amy Machado, Research Manager, Streaming Data Pipeline, IDC. 

"Businesses need to add more streaming use cases, but the lack of developer talent and increasing technical debt stand in the way. Visual interfaces, like Stream Designer, are key advancements to overcoming these challenges and make it easier to develop data pipelines for existing teams and the next generation of developers."

"Data streaming is quickly becoming the central nervous system of our infrastructure as it powers real-time customer experiences across our 12 countries of operations," adds Enes Hoxha, Enterprise Architect, Raiffeisen Bank International. 

"Stream Designer's low-code, visual interface will enable more developers, across our entire organisation, to leverage data in motion. With a unified, end-to-end view of our streaming data pipelines, it will improve our developer productivity by making real-time applications, pipeline development, and troubleshooting much easier."

Stream Designer provides developers with a flexible point-and-click canvas to build pipelines in minutes and easily describe data flows and business logic within the Confluent Cloud UI. 

It takes a developer-centric approach, where users with different skills and needs can seamlessly switch between the UI, a code editor, and a command-line interface to declaratively build data flow logic at top speed. 

It brings developer-oriented practices to pipelines, making it easier for developers new to Kafka to scale data streaming projects faster.

Organisations using Stream Designer enjoy many benefits. 

Instead of spending days or months managing individual components on open-source Kafka, developers can build pipelines with the complete Kafka ecosystem accessible in one visual interface. They can build, iterate and test before deploying into production in a modular fashion, keeping with popular agile development methodologies. There's no longer a need to work across multiple discrete components, like Kafka Streams and Kafka Connect, requiring their own boilerplate code.

After building a pipeline, the next challenge is maintaining and updating it over its lifecycle as business requirements change and tech stacks evolve. Stream Designer provides a unified, end-to-end view to observe easily, edit, and manage pipelines and keep them up to date.

Moreover, pipelines built on Stream Designer can be exported as SQL source code for sharing with other teams, deploying to another environment, or fitting into existing CI/CD workflows. 

Finally, Stream Designer allows multiple users to edit and work on the same pipeline live, enabling seamless collaboration and knowledge transfer.