Introduction

The Challenge

The client faced significant inefficiencies in their supplier data ingestion process, which relied heavily on a stream of unstructured data from various sources. Suppliers submitted product-relevant information in disparate formats such as Excel sheets, emails, and miscellaneous files. Consequently, e-commerce supply chain teamss spent several days manually mapping this data to the formats required by the product engine, causing delays and potential inaccuracies.

The client's primary goal was to revolutionize this process by fully automating the data ingestion system. They aimed to transform the ingestion cycle from a tedious, days-long process into a swift, minute-by-minute operation, thus enabling real-time decision-making and significantly enhancing operational efficiency.

The Solution

To address this complex challenge, our solution was designed to be both highly customizable and scalable. It needed to seamlessly manage various product configurations and efficiently process hundreds of thousands of data points regularly submitted by suppliers. Our goal was also to reduce operational efforts associated with managing such a system.The technology foundation for our solution is built on a robust set of AWS serverless event-based services. This includes:

• AWS Step Functions: Used for managing workflows and dynamically generating configuration files. This allows us to tailor the orchestration of Lambda micro-tasks via a user-friendly interface, making it possible to adjust workflows by simply dragging and dropping components as needed.
• AWS Lambda: Functions as the core processing unit, handling tasks without the need for server management.
• DynamoDB Streams: Provides a highly reliable event-streaming service that captures data modifications in DynamoDB tables in real-time.
• SQS Queues: Manages message queuing to ensure smooth communication between different service components.
• AWS X-Ray: For end-to-end request traceability and operational monitoring.
• API Gateway, AWS Amplify, and AppSync: These fronting services handle interactions and connectivity, enhancing the user experience and integration capabilities.
• Two standout services in our toolkit are DynamoDB Streams For its robustness in event streaming.
• Amazon Bedrock: Utilized for its advanced AI-assisted column-matching capabilities, which significantly enhance data accuracy and efficiency.

By leveraging these advanced technologies, they've been able to meet the complex needs of dynamic and scalable supplier management with reduced operational demands.

Implementation Process

The implementation process took about four months with a team of 5 engineers and 1 business analyst. In close collaboration with the customer, using SCRUM methodology, they completed all the implementation stages in an agreed timeframe. Requirements Analysis: Create comprehensive business requirements documents.Cloud Architecture: Design a cloud architecture for the required software solution.Infrastructure as Code: Script the whole AWS environment (approx. 900 resources) using Terraform. Deployment time to a new environment stage is about 4 minutes.CI Pipeline Engineering: Develop a CI pipeline for automated testing and deployment across multiple stages (test, pre-production, production).

Testing: Local testing of Lambda functions using AWS SAM tool and CI integrated unit and integration testing.

Results Achieved

• Time-to-Market Advantage: The solution provides a substantial advantage over competitors by enabling real-time issuance of insurance quotes, compared to the traditional annual update. This responsiveness has transformed the client's market presence, allowing them to capitalize on immediate opportunities and adapt quickly to market changes.
• Enhanced Operational Efficiency: Automation has minimized manual tasks, allowing staff to focus on higher-value activities, which improves productivity and reduces the potential for human error.
• Cost Efficiency: By adopting a serverless model, the client has significantly reduced overhead costs associated with managing and maintaining static cloud resources, leading to lower operational expenses.

Lessons Learned

This project ventured into unexplored territory with a first-of-its-kind product, facing constantly evolving requirements and numerous uncertainties. Here are the critical lessons we learned about maintaining speed and flexibility in such a dynamic environment:

• Prioritizing Speed and Adaptability: From the outset, it was clear that quick adaptation to changes and rapid development cycles would be crucial. We focused on maintaining high agility throughout the project to respond swiftly to any new challenges or shifts in requirements.
• Implementing Quick Feedback Cycles: To ensure our approach remained on target, we initiated a series of two-week proof-of-concept (POC) cycles. These short cycles allowed us to quickly validate our assumptions using AWS services, providing a fast turnaround on new ideas and immediate testing. This rapid validation process was essential for keeping the project aligned with the client’s evolving needs.
• Streamlining Development with IaC: Once each PoC received customer approval, we swiftly moved the configurations into our Infrastructure as Code (IaC) tools, specifically Terraform. This transition not only streamlined the development process but also prepared us for scaling and replicating the solution efficiently across different environments.
• Continuous Integration and Deployment: Concurrently with the POC testing, our DevOps team worked on establishing robust GitLab CI pipelines and refining the Terraform code for each validated POC. This setup ensured that as soon as a concept was proven, it could immediately be integrated into the ongoing development pipeline, reducing delays and enhancing product integrity.