FinTech Platform for Data Analysis and Financial Reporting on Amazon ECS
Implementation of scalable, secure, and highly available cloud infrastructure for a B2B energy sector FinTech platform using Amazon ECS, RDS, auto-scaling groups, AWS Secrets Manager, SSM Parameter Store, serverless data pipelines, AWS Cognito with Okta for SSO, and CI/CD with GitHub actions.
Problem Statement/ Definition
Our client, a prominent player in the B2B energy sector, faced significant operational hurdles due to their legacy system, which had evolved incrementally over many years. The core issues they encountered included:
– An outdated and slow database that struggled to keep pace with the company’s data demands.
– A cluttered web application interface that complicates user interaction, increasing the likelihood of user errors.
– A disjointed system architecture where data ingestion and reporting functionalities were segregated across different interfaces, hindering efficient data management.
– Inadequate data experimentation capabilities, preventing effective data preparation for ingestion.
– Poor data validation processes within the old application, leading to incomprehensible errors when encountering problematic input data.
These challenges not only hampered the company’s operational efficiency but also impeded their ability to make informed, data-driven decisions.
Proposed Solution & Architecture
To address these challenges and transform our client’s data management and reporting capabilities, we developed a new FinTech platform tailored to the needs of the B2B energy sector. The revamped platform offers:
– A unified interface that seamlessly integrates data ingestion and reporting functionalities, significantly reducing user error and enhancing overall user experience.
– Advanced data validation features that not only detect issues but also suggest potential fixes, enabling users to address data quality proactively.
– A decoupled data analytics architecture where data cleanup, transformation, and ETL (Extract, Transform, Load) processes are isolated from the web application. This separation significantly boosts the system’s data handling capabilities and accommodates larger data volumes more effectively.
– Improved data organization through the segregation of time-series data, its metadata, and user database information. This strategic separation streamlines data processing and enhances the clarity and accessibility of data insights.
The solution architecture is provisioned through Infrastructure as Code using our proprietary ECS solution blueprint, and consists of the following components:
– AWS Cognito for user authentication through customer’s SSO – Okta
– Containerized application layers: Frontend, Backend, GraphQL Hasura middleware
– Amazon ECS cluster to orchestrate deployment and execution of application services
– Auto-scaling group of EC2 instances to host and scale application workloads
– Amazon RDS instance hosting the main database of the platform
– AWS Secrets Manager secret to store main application secrets accessible across all containers
– AWS SSM Parameters to store sensitive (secret) and non-sensitive container-specific parameters
– Application Load Balancer to route and balance traffic flowing to application containers
– S3 Buckets providing storage for platform’s data pipelines
– Lambda functions executing data pipelines
Outcomes of Project & Success Metrics
By implementing these solutions, we have delivered a platform that not only resolves the existing inefficiencies and limitations but also provides a robust foundation for future innovation and growth in data analysis and financial reporting within the B2B energy sector.
TCO Analysis
Platform Development:
– 2 Full-stack developers
– 1 Data Ops / Scientist
– 1 UI/UX designer
– 1 Delivery Manager
Infrastructure Development:
– 1 DevOps x 3 weeks
Maintenance:
– Monthly cost: ~$500
Lessons Learned
Protecting the application web interface and APIs from unauthorized access underscored the need for incorporating security best practices early in the development lifecycle. Tools like AWS WAF and IAM roles proved essential in safeguarding against data breaches.
Industry Vertical
Use case
ISV tools and technology
GitHub – source code repository
SonarQube – for source code scanning
Cypress – for acceptance testing
Docker – for containerization of the app services
Terraform – for IaC
Python Pandas – for data manipulation