Prasanna Anandan is a Senior Vice President and Senior Lead Software Engineer in a Site Reliability Engineering organization at a leading global financial institution, where he supports Market and Country Risk and related reputation-risk functions. Over the course of his career, he has focused on redesigning how complex financial systems detect, validate, and resolve data discrepancies under regulatory time pressure. His work centers on reducing reconciliation latency in high-volume environments by shifting validation from delayed batch cycles to on-demand, user-initiated workflows that preserve auditability and control. In this conversation, Anandan explains how recurring reconciliation failures emerge across interconnected systems and how architectural changes can materially improve reliability, turnaround time, and reporting readiness.
ELLEN WARREN: Prasanna, your work today focuses on improving how complex financial systems detect and resolve reconciliation failures under real-world conditions. How did your career evolve toward redesigning these workflows, and what patterns led you to focus on this problem?
PRASANNA ANANDAN: I came up through the engineering side where you are judged by design quality and development experience aligned with modern programming practices. I worked on improving reliability by building the monitoring foundation using tools such as Prometheus, Datadog, and Grafana, which help teams quickly diagnose system issues and improve the customer experience. Early on, I kept seeing the same pattern: the business logic might be correct, but reliability and workflow design were treated as afterthoughts. In risk environments, that does not work. Month-end and quarter-end deadlines do not care that a job is supposed to run—they care whether the output is ready and traceable when needed. Over time, I shifted my focus from keeping systems running to reducing the time and effort required to detect, validate, analyze, and resolve recurring issues. In regulated environments, that is where reliability and automation have the most impact.
EW: In your role within a Site Reliability Engineering organization supporting Market and Country Risk functions, where do reliability challenges most often emerge in financial risk infrastructure?
PA: Reliability challenges most often emerge at the points where risk, finance, and reference-data systems intersect. Reconciliation depends on multiple upstream feeds and downstream reporting stores, and issues arise when any of those dependencies behave inconsistently during peak cycles. Common patterns include delayed or partial upstream delivery, schema or reference-data drift between systems, API timeouts or throttling under reporting load, and batch runs that appear successful but produce incomplete or stale output due to silent data gaps. I have also worked on introducing a GAIA Cloud-based environment in place of traditional physical data centers. This cloud approach has demonstrated improved reliability, with system availability measured at 98.97 percent. Because data is managed within cloud infrastructure, it is less affected by physical server limitations. When reliability breaks down, the operational impact is immediate: teams lose confidence in the pipeline, and workflows shift from controlled validation to manual investigation.
EW: Financial institutions process large volumes of reconciliation activity across many systems and asset classes. In your experience, where do reconciliation workflows most often begin to break down?
PA: In my experience, reconciliation workflows most often break down when upstream data sources deliver incomplete or inconsistent information. In large financial environments, risk and reporting systems depend heavily on both external and internal data feeds, including providers such as Bloomberg. When these feeds are not fully synchronized, downstream systems still process what they receive, which can introduce partial or outdated records into the pipeline. The issue becomes visible when Market Risk or reporting teams attempt to reconcile outputs from two systems that are expected to represent the same positions or valuations. If one system has received a complete update while another has processed only a partial feed, the reconciliation layer surfaces mismatches that are not business errors but data consistency gaps. I have seen many reconciliation failures originate at this point, where otherwise reliable systems produce conflicting outputs because the underlying data arrived at different levels of completeness or timing.
EW: When you examined the reconciliation lifecycle across systems and teams, what types of delays appeared most frequently, and how did those delays affect reporting timelines during month-end and quarter-end cycles?
PA: When I examined the lifecycle end to end, the most frequent delays were not caused by a single component; they were cumulative. There were scheduler wait times, delays in multi-API data retrieval, post-run wait times for artifacts to become available, and repeated manual intervention when dependencies failed or datasets arrived late. During month-end and quarter-end cycles, these delays compound and cascade. Late availability of reconciliation output pushes resolution and reruns later, which compresses readiness for reporting and approval. This results in two measurable effects: increased reconciliation turnaround time, often extending from a few hours to several business days when delays stack, and significantly higher manual effort as analysts spend time revalidating and re-extracting data while reporting deadlines approach. These delays can also affect regulatory reporting timelines, such as submissions expected by BD5 (the fifth business day of the month).
EW: Many reconciliation environments still depend on scheduler-driven batch jobs. What limitations did you observe in that architecture as reporting volumes and system dependencies increased?
PA: Scheduler-driven batch architectures have predictable limitations as volumes and dependencies increase. They operate within fixed execution windows that do not align with real operational needs, meaning users cannot initiate validation when required but must wait for scheduled runs. Batch pipelines also accumulate latency from every dependency, including API calls, database queries, and upstream feeds. Failures become costly because a single upstream issue often forces a full rerun rather than a targeted revalidation, adding time and pressure during reporting cycles. In addition, batch execution can encourage teams to treat job completion as a proxy for correctness. In risk and reporting environments, the requirement is correctness and traceability, not just completion. When validation is deferred until the end of a batch cycle, errors are identified at the point where tolerance is lowest.
EW: Your solution introduced a user-initiated AI interface embedded within Symphony Messenger, a platform analysts already use to collaborate. What requirements guided the design so the system could meet enterprise security and regulatory control standards?
PA: The design requirements were driven by the need to operate within a regulated environment. The interface had to be usable by business analysts while ensuring that the underlying workflow remained compliant and secure. We anchored the solution within Symphony, an approved enterprise collaboration platform in financial services that supports cryptographic isolation. The bot operates within controlled chat rooms managed by Market and Country Risk users, where membership maps to entitlements and access control is enforced at the collaboration layer. The bot responds only within these rooms using a secure room key configured through encrypted certificates. In addition, the system generates a full audit record of execution, including who initiated the request, the parameters used, the dataset and time period validated, the result returned, and the execution status.
EW: The reconciliation engines themselves remained in place. Which architectural changes allowed execution speed to improve while preserving the existing validation and control framework?
PA: Execution speed improved because we changed the initiation and orchestration layer rather than rewriting the reconciliation engines. The engines and validation rules remained intact to preserve existing control frameworks and business-approved logic. The bot acts as a user-initiated orchestration gateway, allowing the same underlying services to be triggered on demand instead of waiting for scheduled execution. On the backend, the workflow reduces redundant data pulls by reusing reference datasets within defined freshness windows, avoids unnecessary full reruns by enabling targeted validations, and returns results in stages. Users first receive immediate status and exception counts, followed by detailed drill-down when needed. This approach preserves validation integrity while significantly reducing the time from request to result.
EW: The implementation reduced reconciliation resolution time from multiple business days to under ten minutes for routine tasks. Which design decisions had the greatest influence on that improvement?
PA: The primary driver of improvement was removing idle wait time and replacing it with on-demand validation. In the legacy model, much of the timeline was spent waiting for scheduler cycles, long-running scripts across multiple APIs, and reruns after dependency failures. By enabling analysts to initiate reconciliation directly through Symphony with validated parameters, we significantly reduced that latency. The second driver was output design. Analysts did not need large batches of artifacts to begin triage; they needed immediate signals, such as status, exception counts, and clear identification of discrepancies, followed by controlled drill-down. The third driver was the introduction of stability guardrails, including concurrency controls, timeouts, and predictable failure handling, to prevent load spikes and maintain system stability during peak periods. As a result, routine checks can now be completed in under ten minutes under normal conditions, with reduced manual effort and faster reporting readiness.
EW: In 2024 you received the Outstanding Interdisciplinary Research Award from the International Conference on Convergence Technology in Artificial Intelligence and Robotics. Which aspects of that project required collaboration across different technical or operational disciplines?
PA: Collaboration was essential because the solution operated at the intersection of business workflows and technical automation. Market Risk Middle Office teams defined what reconciliation correctness means across different asset classes, including government bonds, municipal bonds, ETFs, index funds, and mutual funds, and identified which exceptions were material versus informational. Security and platform teams ensured alignment with cryptographic isolation and validated that the bot could operate without expanding data exposure. Compliance and governance stakeholders required auditability, role-based access control, and defensible logging. At the same time, production reliability considerations required performance limits and failure-mode controls to ensure that the on-demand model did not introduce instability during reporting peaks.
EW: After implementing this architecture, where do you see the most practical opportunities to apply similar workflow design principles across other risk or reporting processes?
PA: The most practical opportunities are in workflows related to principal risk, reputational risk, and consumer and community banking. This includes reconciliation and data-quality controls upstream of reporting, break-management workflows involving repeated exception triage, and risk reporting checks where teams need immediate confirmation that datasets are consistent and valid before publication. The principle is transferable: preserve the validated core engines and control frameworks, but redesign the initiation layer so validation begins when discrepancies appear rather than when a scheduler runs. Success should be measured through reduced turnaround time, lower manual effort, faster exception resolution, and maintained or improved audit completeness.
