PAYONE, a leading payment service provider (PSP, FinTech) specializing in B2B SaaS solutions, offers merchants the ability to integrate various payment options seamlessly. Their services extend beyond mere transaction processing to include advanced fraud prevention and multichannel checkout systems.
As a senior software engineer, software architect and tech lead, I had the unique opportunity to drive the formation of PAYONE’s sole remote team from the ground up (2018).
This position came with its set of challenges and advantages, particularly in bridging the communication gap with in-office teams accustomed to face-to-face interactions and navigating the autonomy we possessed in shaping our work processes.
Our main responsibility lay in optimizing the fraud prevention service, a critical function that decides the legitimacy of transactions on the platform.
The Problem
The deployment process at PAYONE was cumbersome and outdated, a relic from the days of a monolithic platform architecture. The process grew in size of complexity over time, trying to foresee and prevent any possible incident. This process severely hindered our ability to efficiently update our fraud prevention service, impacting our responsiveness to new threats and operational efficiency. Key challenges included:
- Complex Coordination: Deployment required coordination and meetings between our development team and the operations team, slowing down the release of updates.
- Manual Database Changes: Database updates were particularly problematic, requiring manual execution by the operations team to apply SQL changes, a process prone to errors and delays.
- Infrequent Deployments: The existing system allowed for updates only once every two weeks or less, a pace inadequate for the fast-moving digital payment landscape. Our team often forgot about the changes that were not deployed yet, leading to wrong assumptions and incidents.
- Operational Silos: A significant barrier was the organizational separation between our development team and the operations team, each with different priorities and reporting lines, complicating any process overhaul.
The Solution
To address these challenges, we implemented several strategic initiatives:
- Visibility of Pending Work: We introduced a “to be deployed” tracking column on our Jira Board, making it immediately evident which updates were queued for release. This transparency helped in recognizing the backlog of enhancements and bug fixes awaiting deployment, underscoring the inefficiency of the current process to the broader team and management.
- Work-in-Progress Limits: By setting a limit on the number of updates awaiting deployment, we created a natural urgency to facilitate more frequent deployment meetings. This not only encouraged regular interactions with the operations team but also focused discussions on how to streamline the deployment process.
- Automated Database Migrations: We adopted Liquibase, an automated database migration tool, which significantly reduced the risk of errors in applying database updates and streamlined the overall deployment process.
- Increased Deployment Frequency: Moving to a weekly deployment schedule forced us to refine and streamline our deployment processes, making them more efficient and less time-consuming.
- Service Decoupling: We migrated our service to independent virtual servers, allowing for more flexible and isolated deployments. This step significantly reduced the dependency on shared infrastructure and minimized the risk of cross-service disruptions.
Outcomes
The overhaul of the deployment process yielded significant improvements across several dimensions:
- Efficiency Gains: Transitioning to weekly deployments, we maintained operational workload while significantly increasing the frequency of updates. This change led to more timely enhancements and fixes to our fraud prevention service, directly benefiting our clients.
- Improved Oversight: The visibility into pending deployments heightened awareness among the development team and management, fostering a culture of continuous improvement and accountability.
- Risk Reduction: Smaller, more frequent updates reduced the complexity and risk associated with each deployment, improving system stability and trust between the development and operations teams.
- Faster Feedback Loop: With more regular updates, we could iterate on features and fixes more rapidly, responding to customer needs and system performance insights with greater agility.
- Enhanced Collaboration: The process improvements fostered a better understanding and collaboration between the development and operations teams, breaking down silos and aligning goals.
Lessons Learned
Our experience at PAYONE underscored the importance of visibility in bottlenecks, stakeholder inclusion in process changes, and incremental steps toward ideal deployment practices. Despite organizational constraints, we demonstrated that significant process improvements are feasible and can lay the groundwork for even more ambitious goals, such as continuous deployment.
These were the key success drivers in this story:
- Visibility as a Catalyst for Change: The first step toward addressing a problem is making it visible. By making unfinished work and bottlenecks transparent, we increased our awareness and highlighted inefficiencies to management. This visibility acted as a powerful motivator for change, underscoring the principle that you cannot improve what you do not measure.
- Stakeholder Inclusion is Key: One of the most critical lessons learned was the importance of including all relevant stakeholders in the process of change. In our case, despite the initial resistance, the operations team became our allies in transformation. Their insights and concerns were invaluable in shaping a deployment process that met both teams’ needs. This experience reinforced the idea that sustainable change requires consensus and collaboration, not just top-down directives.
- Incremental Changes Lead to Major Wins: When faced with organizational constraints, it’s tempting to push for transformational changes. However, our journey at PAYONE demonstrates the power of incremental improvements. Each small step in the right direction paved the way for the next, building confidence and trust among all stakeholders. This gradual approach not only made the changes easier to follow, but also allowed for adjustments based on feedback and evolving needs.
- The Journey Towards Continuous Deployment: While we did not achieve continuous deployment due to various constraints, our efforts moved us significantly closer to that ideal. This experience highlighted the importance of adapting to the organization’s current state while steadily pushing the boundaries. It’s a reminder that even in less-than-ideal conditions, progress is possible, and each step forward lays the groundwork for future advancements.
Conclusion
The PAYONE case study exemplifies how teams can navigate organizational constraints to significantly improve deployment processes, enhancing both work experience and value delivery. Our journey, marked by gradual, safe improvements and stakeholder engagement, serves as a blueprint for organizations aiming to refine their deployment strategies while accommodating evolving needs and perspectives.
