Just because a system is important and widely used doesn’t mean it’s perfect — just look at the New York City subway.
The same is true of Apache Kafka, a data-streaming platform used by 80 percent of Fortune 100 companies to do things like process and analyze sales data in real time, monitor transactions for fraud and track the movement of goods throughout a supply chain.
Like any technology, Kafka has its limitations — though for Stéphane Derosiaux, the co-founder and chief technical officer of Conduktor, these limitations are just opportunities.
Conduktor’s mission is to make Kafka more accessible and manageable for modern enterprises, saving them both time and money. Its solutions include an intuitive user interface, streamlined troubleshooting and the ability to monitor operations for errors.
Already, more than 150,000 developers rely on Conduktor’s solutions — and its latest launch, Conduktor Gateway, was inspired by ongoing conversations with this user base about their experiences and business needs.
Gateway introduces a wide range of new functionalities in Kafka, including a means of simulating and testing application problems, a tool for setting guardrails around tasks users can perform and a plugin for end-to-end encryption.
However, Gateway is just that — a threshold toward a new way of working — and feedback from customers will guide the development of future features and improvements, Derosiaux told Built In.
Read on to find out more about Conduktor’s user-centric approach, technical vision and strategies for building a collaborative culture.
Why did Conduktor need to build this product?
We are building products for modern enterprises using Apache Kafka as their data backbone.
Apache Kafka is an open-source project designed to facilitate real-time data exchange between all IT teams and applications at an enterprise level. We have built an enterprise product to ease the jobs of developers and operators working with Kafka on a daily basis. By providing a user interface that covers all their needs and use cases — data exploration, security, governance — we help them be more efficient and deliver more quickly.
However, while an interface is great, it depends on the underlying technology installed on the customer side: Apache Kafka, in this case.
Also, an interface is for humans, not applications. It was therefore impossible for us to provide new features that would benefit both users and applications simultaneously. This is why we built a new product on the infrastructure layer, closely connected with and enveloping Apache Kafka itself: Conduktor Gateway.
This product allows us to tell a more complete and cohesive story: An organization doesn't only want to control their users interacting with Apache Kafka; they also want to control their applications that interact with it — in terms of data security, permissions, audits, etcetera.
An organization doesn't only want to control their users interacting with Apache Kafka; they also want to control their applications that interact with it — this is why we built Conduktor Gateway.”
What tools or technologies did your team use to build the product and why?
I work with talented engineers, guiding them towards efficiently achieving company objectives while fostering a culture of innovation and learning. This new product is built in a recent version of Java because it’s well-known in the engineering world and it’s performant. This is crucial since this product is on the critical path of the enterprise data flow, handling large volumes of data — up to gigabites per hour for big customers!
By leveraging Java, which is also used in Apache Kafka and its ecosystem, we ensure compatibility with third parties’ libraries as well.
What obstacles did you encounter along the way? How did you successfully overcome them?
This new product started as a proof of concept and required significant upfront effort to showcase its value. Performance and stability are key factors for this product.
In a clever move, we decided to use our cloud and integrated our new product into our existing one, granting thousands of users free access. This allowed us to test and improve its stability, resilience and performance at scale.
In a clever move, we integrated our new product into our existing one, granting thousands of users free access. This allowed us to test and improve.”
How did you keep team members motivated and aligned throughout the product development process?
We emphasize building the product with design partners — your future customers — in mind. Engage with users who are eager to contribute and want to see your idea succeed. Seek their opinions, share mock interfaces and encourage close collaboration between your engineering and product teams and these design partners. This ensures a deep sense of purpose and meaningful impact.
Lastly, it's important to communicate both successes and failures, as well as positive feedback. Share praise and positive comments from meetups, conferences and sales calls within your internal communication channels like Slack.
What teams did you collaborate with in order to get this across the finish line? What strategies did you employ to ensure that cross-functional collaboration went smoothly?
This product represents a significant strategic investment for the company, and collaboration with all teams has been crucial. A product is an ongoing story until it's deprecated.
Working closely with engineering teams has been a priority, including holding regular one-on-ones, helping them navigate technical decisions and addressing any roadblocks or unknowns that arise. It's essential to immerse yourself as a user, testing the product, reading and contributing to documentation and staying involved in the code. Being an active part of the team is particularly critical during the project's early stages to lay a solid foundation.
Collaboration with the product team is vital for prioritization. We utilize a Reach, Impact, Confidence and Effort model and leverage tools like Productboard to visualize and track priorities. Empowering the sales team to understand the product's challenges, determine pricing and engage potential customers has also been instrumental.
Finally, companywide all-hands meetings have been effective in creating awareness throughout the organization, enabling everyone to understand the impact of the new product and encouraging a culture of collaboration and idea generation.
When you think of other companies in your industry, how does Conduktor compare when it comes to how you build and launch new products?
I would describe the state of our ecosystem — Kafka and real-time data processing — as one of rapid growth and excitement. It's a relatively new domain, and most enterprises are still in the process of adopting it to stay competitive in their fields.
Launching a new product is a complex endeavor. It’s important to find the right timing, as launching too late may indicate that too much time has already been invested. Embracing an iterative approach and accepting the launch of an imperfect product is crucial. We listen to our initial users, make adjustments to the UI, enhance the user experience and keep the momentum going.
