We are joined by Sierrah Coleman, a Technical Product Manager whose career is defined by impactful innovations and strategic leadership at organizations like Indeed, Cisco, BrainPOP, and Jump Ramp Games. Known for leveraging data-driven approaches and translating complex technical concepts into meaningful business outcomes, Sierrah has driven projects that increased customer retention, enhanced machine learning systems, and spearheaded database migrations supporting exponential user growth. Let’s explore her journey, decision-making philosophies, and lessons learned from navigating the intricate world of product management.
When leading the development of the AI-powered query-relevance scoring feature at Indeed, what specific technical or organizational challenges did you face, and how did you overcome them?
Sierrah Coleman: Developing the AI-powered query-relevance scoring feature involved two major challenges: securing organizational buy-in and addressing technical complexity. Convincing the down-funnel ranking teams of this feature’s value required demonstrating its potential impact. I created a proof-of-concept that provided statistically significant evidence of how this feature could enhance ranking for the business and improve outcomes for end users. This helped align stakeholders and ensured the feature’s integration into later-stage ranking models.
From a technical perspective, assigning a numerical value to the relevance of search results retrieved via natural match and blending it with other search retrieval strategies was complex. I collaborated with data scientists to develop algorithms that evaluated relevance scores and ensured seamless integration into existing systems. This rigorous approach resulted in a feature that significantly improved query relevance and supported better ranking outcomes.
At BrainPOP, you led a database refactor and migration project that supported 3x user growth. How did you approach designing a scalable architecture, and what were the critical trade-offs you had to make?
Sierrah Coleman: To design a scalable architecture, we transitioned to a relational database system that unblocked critical features and enabled greater flexibility. This approach allowed us to support more diverse product offerings, such as expanding the range of products and subscriptions available to users, which attracted a broader audience. Additionally, it enhanced our ability to query customer orders and accounts, improving troubleshooting and customer support.
The critical trade-off was balancing the immediate need for scalability with the complexities of migration. By prioritizing phased rollouts and thorough testing, we ensured minimal disruptions while achieving a scalable solution that supported rapid user growth and improved operational efficiency.
During your tenure at Kenna Security (now Cisco), you increased customer retention by 12% and boosted annual revenue by 5%. What role did competitive analysis and user feedback play in shaping these outcomes?
Sierrah Coleman: Competitive analysis and user feedback were pivotal in shaping these outcomes. We conducted a SWOT analysis of competitor offerings, uncovering gaps in our product’s capabilities. Simultaneously, user feedback highlighted pain points and unmet needs.
Using this data, we prioritized features like enhanced reporting dashboards, which allowed clients to intuitively drill down into risk scores for groups of assets. We also introduced more customizable and tailored data exports, as well as compliance-friendly audit logging and reporting. These features addressed critical user needs, resulting in increased client satisfaction and retention, which directly contributed to the 12% retention increase and 5% revenue growth.
The offline simulation platform for predicting recommender system performance at Indeed reduced evaluation timelines by 33%. Can you share the key metrics and methodologies you used to measure success?
Sierrah Coleman: The success of the offline simulation platform was measured using three key metrics: evaluation speed, predictive accuracy, and user engagement post-deployment. Evaluation speed improved by 33%, reducing iteration cycles for recommender systems. Predictive accuracy was measured by comparing offline results with live A/B testing outcomes, ensuring less than a 5% deviation.
We adopted methodologies such as synthetic data generation to simulate diverse user scenarios. This approach provided a reliable testing environment, allowing for faster and more precise refinements to our recommender systems.
What lessons from managing the launch of two F2P mobile games at Jump Ramp Games have stayed with you, especially in terms of navigating the trade-offs between creative freedom and business objectives?
Sierrah Coleman: A key lesson was the importance of aligning creative teams with business goals early in the development process. By establishing clear KPIs like user acquisition cost (UAC) and lifetime value (LTV), we created a shared framework for evaluating design decisions.
One trade-off involved balancing innovative gameplay mechanics with monetization strategies. To address this, we ran iterative playtesting sessions, analyzing player engagement metrics and in-app purchase behaviors. This approach allowed us to maintain creative freedom while ensuring the games met revenue targets.
The transition from SMTP to REST APIs for Sparkpost boosted email throughput by 22% at Indeed. What drove this decision, and how did you manage the transition without disrupting existing workflows?
Sierrah Coleman: The decision was driven by the need for scalability and improved error handling. SMTP lacked the flexibility to handle dynamic routing and monitoring at scale, which REST APIs offered.
To manage the transition, we developed a dual-stack system that allowed both protocols to run simultaneously during migration. This ensured uninterrupted email delivery while we tested and optimized the REST API implementation. Extensive documentation and training for engineering teams minimized disruptions, resulting in a seamless transition and a 22% increase in throughput.
At BrainPOP, your audit of the technical landscape highlighted areas for cost savings worth 4% of the annual budget. What tools and frameworks did you use to conduct this analysis, and what were the biggest surprises?
Sierrah Coleman: My approach centered on interviews with department heads across various teams, including tech/R&D, customer success, product design, product management, sales, marketing, and finance. These interviews helped me understand the internal tools they were using and the requirements each team had. Using this information, I conducted a comprehensive audit of the internal tool ecosystem to assess the tools in use versus those that were necessary given the database re-architecture project.
I kept a detailed record of tools the company was purchasing, noting instances of redundant subscriptions or overlapping functionalities. The biggest surprise was discovering multiple subscriptions for the same tool and significant overlap in tools serving similar functions. By consolidating resources and decommissioning underutilized assets, we achieved a 4% cost savings .
While working on ML-based recommender systems at Indeed, you collaborated with data scientists to develop implicit relevance metrics that improved explicit user ratings by 9%. How did these metrics influence the experiments, and what role did they play in enhancing overall system performance?
Sierrah Coleman: Developing implicit relevance metrics was a pivotal step in refining our ML-based recommendation systems. These metrics served as proxies for user satisfaction, capturing nuanced behavioral patterns such as dwell time, time-to-value metrics, and engagement signals. These were critical in shaping experiments designed to improve match relevance.
Collaboration with data scientists ensured the metrics were robust and grounded in real user behavior. We used these metrics as success criteria during experimentation, iterating on recommenders to better align with user expectations. Over time, the improvements in implicit metrics correlated with a 9% increase in explicit relevance ratings provided by users, validating the value of our approach.
By incorporating these metrics into the evaluation framework, we not only enhanced the precision of our recommendations but also reduced the feedback loop time, enabling faster iterations. This iterative cycle ultimately resulted in a system that delivered more personalized and relevant results, driving better user engagement and satisfaction.
During your time at Kenna Security (now Cisco), the launch of an auditing feature secured upsell opportunities worth $300,000+ and a $400,000+ contract renewal. What was your approach to identifying and pitching this feature to key stakeholders?
Sierrah Coleman: This feature was identified in partnership with our customer experience team, who highlighted client feedback about compliance challenges as a significant pain point. Competitive analysis also revealed that offering this feature would position our product as a leader in the market and enhance our competitive edge.
To pitch the feature, I developed a comprehensive business case detailing the ROI, as well as the development and implementation plan. This enabled me to successfully secure buy-in from leadership and stakeholder teams. As a result, the feature was successfully launched, driving upsell opportunities and supporting contract renewals.
With PSPO I and II certifications, you have a strong commitment to agile methodologies. How do you ensure that agile practices remain effective as team structures and project scopes evolve?
Sierrah Coleman: The PSPO I and II certifications have provided me with valuable insights into maximizing product value within the Scrum framework. These certifications reinforce the importance of keeping agile principles and product value top of mind as project scopes evolve.
To ensure agile practices remain effective, I conduct regular retrospectives and continuously evaluate how practices like sprint planning and backlog refinement can be adapted to meet the team’s, business’s, and users’ needs. Agile is designed to be flexible and iterative, and these certifications enable me to guide teams in adapting practices that optimize efficiency and drive innovation. This focus on product value and adaptability helps maintain alignment and effectiveness as team structures and project priorities change.
