As businesses continue to scale and modernize their digital infrastructure, distributed storage and cloud-native systems are becoming critical pillars for innovation. In this interview, Harsh Desai, a leading specialist in cloud-native storage architecture, shares his expertise on overcoming the challenges of distributed systems, the evolving role of Kubernetes in storage orchestration, and the future of AI-driven optimization in complex infrastructure environments. With a career dedicated to pushing the boundaries of scalable, resilient storage solutions, Desai offers a forward-looking perspective on where the industry is heading and what technologies will define the next generation of cloud infrastructure.
Q: Can you walk us through one of your most impactful projects in distributed storage — what problem were you solving and what was your role in its success?
One of my most impactful projects was leading the Autopilot project at Portworx, which addressed the critical challenge of automated storage capacity management and workflow automation in Kubernetes clusters. During my tenure as a Member of Technical Staff (April 2017 – June 2021), as Portworx’s customer base grew, the product faced significant issues like cluster instability, poor debuggability, and increasing development overhead due to growing demand. I became the inventor and sole maintainer of the Autopilot product, which automated customer infrastructure monitoring and workflows. My role involved not only the architectural design but also overseeing its implementation, which was so unique it resulted in five granted US patents. This project significantly enhanced Portworx’s ability to handle increasing demand and maintain stability at scale, directly contributing to the company’s consistent double-digit year-over-year growth and ultimately its acquisition by Pure Storage for $370 million.
Q: What’s a decision you made in a past architecture design that you’re especially proud of, and why did it matter?
I am particularly proud of my role as the original architect of the Portworx Control Plane during its early startup phase (2017-2020). The crucial architectural decision was to decouple core components and introduce a modular, microservice-oriented architecture. This was a “game-changer” because it enabled Portworx to scale innovation at a record-breaking pace, significantly slashing operational overhead, accelerating deployments, and delivering highly resilient, production-grade software. This foundational architectural design was critical for Portworx’s transformation from a fledgling startup to a powerhouse with over 100 enterprise customers. Even today, Pure Storage continues to build incrementally on this very foundation, highlighting its lasting impact and importance to the company’s current and future success.
Q: Have you ever introduced a novel approach or breakthrough in storage or Kubernetes architecture that significantly improved system performance or reliability?
Yes, a key breakthrough was the telemetry-driven proactive issue detection system I led and architected as a Senior Engineering Manager at Portworx. The problem was the unsustainable growth in customer support overhead as our customer base expanded. My approach involved collecting telemetry data from customer systems, running analytics to proactively detect potential issues, and automatically creating support tickets with potential resolutions. This system significantly reduced customer support cases by approximately 50 per month, saving the business unit thousands of dollars and hundreds of hours of engineering time. This novel method drastically improved system reliability and uptime by identifying and addressing problems before they escalated, while simultaneously boosting support efficiency and overall customer satisfaction.
Q: What’s a particularly tough challenge you’ve faced in scaling a storage system in Kubernetes — and how did you overcome it?
A particularly tough challenge I faced in scaling a storage system in Kubernetes was at Portworx between 2019 and 2020. As the customer footprint rapidly expanded, the Portworx Product struggled to keep up with growing demand, leading to severe cluster instability, poor debuggability, and increasing development overhead on an aging codebase. Frequent software emergencies further hampered the engineering team’s ability to innovate. To overcome this, I led numerous supportability releases for Portworx Enterprise, specifically focusing on stability at scale and handling negative scenarios. I implemented a structured approach by conducting weekly product gap meetings, gathering customer feedback, prioritizing critical issues in product releases, and planning larger re-architecture items for the roadmap. This combination of deep technical acumen, strategic foresight, and a customer-first mindset was crucial in stabilizing the platform and enabling Portworx’s consistent double-digit year-over-year growth.
Q: Can you share an example of how your leadership influenced the success of a team working on cloud-native infrastructure?
As Senior Engineering Manager at Pure Storage (Portworx) since June 2021, I lead a globally distributed team of over 20 engineers across the U.S. and India, responsible for the Portworx Enterprise Control Plane. One significant example of my leadership’s influence is in driving strategic Artificial Intelligence (AI) initiatives that directly impact developer productivity and operational efficiency. I introduced processes and tools that mandate the use of AI for test code generation, which boosted our testing coverage by 30% in just two months. Additionally, I spearheaded the use of AI for analyzing millions of log lines and customer data to automatically triage and identify issues, saving the team hundreds of hours of developer time monthly. These initiatives have not only enhanced our development velocity but also significantly improved the reliability and quality of our cloud-native infrastructure, fostering a culture of innovation and efficiency within the team.
Q: What’s one product, system, or feature you helped ship that you believe moved the needle for users or the industry?
The Kubernetes Control Plane for Portworx Enterprise is undoubtedly a product I helped ship that moved the needle for users and the industry. Early in Portworx’s history, I spearheaded its development and became an open-source contributor to the Kubernetes community, building the foundational driver that integrated Portworx into the Kubernetes ecosystem. This was a critical achievement because, without these early contributions, Portworx would not have gained the initial momentum necessary to fuel its future growth. This fundamental integration directly led to securing major enterprise clients like T-Mobile, American Express, Comcast, IBM, and HSBC, generating over $10 million in annual revenue from these accounts alone. It positioned Portworx as a leader in cloud-native storage, ultimately leading to its acquisition by Pure Storage.
Q: Have you contributed to any open-source projects in the Kubernetes or distributed systems ecosystem? What was your contribution and its impact?
Yes, I have made several contributions to open-source projects within the Kubernetes and distributed systems ecosystem. As a Member of Technical Staff at Portworx, I contributed to projects such as Stork, OpenStorage, and the Autopilot API, among others. More broadly, I became an early open-source contributor to the Kubernetes community itself by developing the foundational driver that integrated Portworx into the Kubernetes ecosystem. This integration was absolutely critical for Portworx’s early adoption and established its presence as a key player in cloud-native storage. My efforts helped lay the groundwork for how Portworx would interact with and extend Kubernetes functionalities, which was essential for the company’s initial momentum and subsequent success in the industry.
Q: Tell us about a time you had to advocate for a technical direction or architecture decision — what was at stake, and how did it turn out?
During my time as the original architect of the Portworx Control Plane, I had to strongly advocate for implementing a modular, microservice-oriented architecture. At stake was the company’s ability to effectively scale, innovate, and maintain product stability in a rapidly evolving cloud-native landscape. The alternative was a more monolithic structure, which would eventually lead to bottlenecks and technical debt. My advocacy involved not only designing this architecture but also illustrating its long-term benefits in terms of accelerated deployments, reduced operational overhead, and enhanced resilience. This technical direction proved to be a game-changer for Portworx. It enabled us to scale innovation at a record-breaking pace, transformed Portworx from a fledgling startup into a powerhouse with over 100 enterprise customers, and was instrumental in the company’s eventual acquisition by Pure Storage for $370 million [71,Cluster rebalance using user defined rules”, “Mount path management for virtual storage volumes in a containerized storage environment”, and “Usage-based Restore Prioritization”. The fact that these patents have been cited 21 times by other scholars indicates that they are recognized contributions that have informed and guided subsequent research and development in the industry. This direct contribution to intellectual property and the academic discourse ensures a sustained impact on the foundational principles and advanced practices within cloud-native and distributed storage systems.
Q: Looking back on your career so far, which moment or project are you most proud of — and what made it so meaningful to you?
Reflecting on my career, the moment I am most proud of is witnessing the Portworx business unit’s revenue grow from $0 to over $50 million annually. I joined the team when it had no revenue, and as a Senior Engineering Manager, I have had a critical role in its growth, with our annual revenue reaching over $50 million in 2024, a 400% increase from 2020. This includes a record-breaking quarter exceeding $30 million, directly attributable to product capabilities developed by my team. This journey, from a nascent startup phase to becoming a significant contributor to Pure Storage’s multi-million dollar data platform, has been incredibly meaningful. It reflects not just technical achievements but also the impact of my leadership, strategic decision-making, and ability to build and guide high-performing teams in the cloud-native space.
