DevSecOps Inc. responded to the customer’s need to modernize cloud infrastructure supporting their legacy FinTech platform, and implemented a scalable, secure, and highly available cloud infrastructure based on Amazon ECS, with RDS database, auto-scaling groups, secrets management using AWS Secrets Manager and SSM Parameter Store, serverless data pipelines, AWS Cognito integration for SSO authentication through clien’t corporate Identity Provider – Okta, and CI/CD integration based on GitHub actions.
Problem Statement
Our client, a prominent player in the B2B energy sector, faced significant operational hurdles due to their legacy system, which had evolved incrementally over many years. The core issues they encountered included:
- An outdated and slow database that struggled to keep pace with the company’s data demands.
- A cluttered web application interface that complicates user interaction, increasing the likelihood of user errors.
- A disjointed system architecture where data ingestion and reporting functionalities were segregated across different interfaces, hindering efficient data management.
- Inadequate data experimentation capabilities, preventing effective data preparation for ingestion.
- Poor data validation processes within the old application, leading to incomprehensible errors when encountering problematic input data.
These challenges not only hampered the company’s operational efficiency but also impeded their ability to make informed, data-driven decisions.
Our Proposed Solution
To address these challenges and transform our client’s data management and reporting capabilities, we developed a new FinTech platform tailored to the needs of the B2B energy sector. The revamped platform offers:
- A unified interface that seamlessly integrates data ingestion and reporting functionalities, significantly reducing user error and enhancing overall user experience.
- Advanced data validation features that not only detect issues but also suggest potential fixes, enabling users to address data quality proactively.
- A decoupled data analytics architecture where data cleanup, transformation, and ETL (Extract, Transform, Load) processes are isolated from the web application. This separation significantly boosts the system’s data handling capabilities and accommodates larger data volumes more effectively.
- Improved data organization through the segregation of time-series data, its metadata, and user database information. This strategic separation streamlines data processing and enhances the clarity and accessibility of data insights.
Solution Infrastructure Components
The solution architecture is provisioned through Infrastructure as Code using our proprietary ECS solution blueprint, and consists of the following components:
- AWS Cognito for user authentication through customer’s SSO – Okta
- Containerized application layers: Frontend, Backend, GraphQL Hasura middleware
- Amazon ECS cluster to orchestrate deployment and execution of application services
- Auto-scaling group of EC2 instances to host and scale application workloads
- Amazon RDS instance hosting the main database of the platform
- AWS Secrets Manager secret to store main application secrets accessible across all containers
- AWS SSM Parameters to store sensitive (secret) and non-sensitive container-specific parameters
- Application Load Balancer to route and balance traffic flowing to application containers
- S3 Buckets providing storage for platform’s data pipelines
- Lambda functions executing data pipelines.
Outcomes
By implementing these solutions, we have delivered a platform that not only resolves the existing inefficiencies and limitations but also provides a robust foundation for future innovation and growth in data analysis and financial reporting within the B2B energy sector.