Customer infrastructure was hosted from one datacentre, with inadequate redundancy and resilience when weighted against their growing customer base and with that the increasing demands on performance, availability, and security. The nature of their business is such that they need to store millions of small files (digital receipts), and this was causing issues for their storage tier.
The business also demanded the highest standards for security, great performance, high availability, scalability, and speed to market, and these goals were becoming difficult to achieve with in-house resources. It was also clear that a partner was needed to help with this digital transformation and to drive that continuous transformation. ETNetwork reached out to Opsio because of their status as an AWS Advanced Consulting Partner and their unique managed services offering and expertise.
The goal had to be to secure and grow the ETNetwork business, and to use the AWS global network and their Nordic presence with datacentres in Sweden to achieve this.
- Automate system operations
- Improve visibility and security
- Scale without Infrastructure constraints
- Get Started quickly and pay as you go
- Leverage fully managed services by AWS Partner
Infrastructure as code
As per Opsio’s strong culture of DevOps, everything we created for the customer was deployed using CloudFormation. We achieved a flexible and complete infrastructure by developing separate CloudFormation templates for the core infrastructure, another for the continuous integration/continuous deployment pipeline for Code Commit and Code Build (covered next), and lastly a template for the tasks running in AWS’s managed Container service, ECS.
As with all Infrastructure as Code deployments, the infrastructure can be versioned, and controlled with all the tools available to application developers. This results in a consistent and reliable deployment every time.
After discussions between ETNetwork and Opsio, it was decided that in order to maintain the highest levels of security, the new Aurora database instance inside the new AWS cloud platform should be hosted on a private subnet, and therefore not accessible from outside of the Virtual Private Cloud (VPC). This meant that a continuous SQL logfile synchronization from the old database to the new Relational Database Service (RDS) hosted database would not be possible. This resulted in a more traditional export and import task, with additional scripts to ensure every last bit of data was captured and copied to the new environment right at the time of migration.
During the course of work to deploy the new cloud infrastructure and migration tasks, it was discovered that so-called secrets, meaning things like passwords and other data which attackers might use to compromise a part of the platform. Opsio worked with ETNetwork and their developers to migrate the secrets into the AWS parameter store, a part of the Systems Manager suite, and have these accessed as part of the CodeBuild step in the CI/CD pipeline.
Scalability and Reliability
As mentioned, a key metric for success was a performant and reliable platform which would handle the growth that ETNetwork and their Kwick product continues to experience. Through a blend of technical solutions and a well managed cloud deployment, these benefits and business objectives have been achieved, resulting in every piece of the ETNetwork platform being elastic, infinitely scalable and secured according to best practices. There are no more limits, and that results in a happy client, and a great partnership.
Opsio designed a solution whereby the client applications would run within Docker containers, and for those containers to be managed and maintained by AWS managed Docker service, Elastic Container Service, or ECS.
With this in place, developers can trigger new code to be built, tested, deployed and migrated through the various testing and staging environments, just by committing into a code repository as then normally would.
SSL Mutual Authentication
Another key deliverable for the ETNetwork platform was the use of SSL Mutual Authentication in specific cases, which is not supported by AWS Application Load Balancers (ALB) by default – a technical solution to this had to be found.
This requirement was satisfied by deploying a separate CloudFormation stack to ECS Fargate; the AWS service which hosts docker containers without the need to provision and manage the underlying virtual servers hosting the containers. A Network Load Balancer (NLB) was placed in front of the Fargate resources, and SSL connections were routed in this way, all the way to a container which would handle the SSL traffic itself and verify the clients.
Systems Manager (SSM)
SSM parameter store is like “salt in our food”. This feature removes the risk of exposing DB passwords and other sensitive parameters we would like to use in our SSM Documents by integrating AWS KMS service. This is a small component of SSM but an essential one without which the service will be incomplete.
Security Best Practices (What’s ON)
- Applications: All necessary application configurations (service endpoints, DB endpoints etc) are stored in parameters store with encryption enabled. These application configurations are made available to the applications during the build stage which makes these configuration parameters extremely secure
- Data in transit: The application deployment strategy ensures protection to data in transit by ensuring the following recommended protection controls
- HTTP/HTTPS traffic (web applications)
- HTTPS offload (web applications to ELB)
- SSH Traffic (Use SSH using non-privileged user accounts)
- Data at rest
- Encryption of data at rest
- S3 versioning
- Database snapshots
- Restrictive IAM permissions with encryption keys
Policies and configuration
We have enabled delegation of access to users and services through IAM Roles and temporary security credentials. Some of the policies implemented are:
- Cross Account Access, Delegation of privileges through IAM Groups & Roles
- Access to EC2s & Other services in form of IAM Roles & Policies
- Parameter Store to store secure credentials & configs
- Web Application Firewalls: Opsio has provisioned the service of Web Application Firewall for the environment.
- Storing & Managing Encryption Keys: All access & secret access keys are stored in AWS KMS for increased security. These credentials are accessed and validated during application or service usage/runtime
- Accidental Deletions: Accidental deletion of databases & other compute services is enabled which protects these services against accidental deletion or termination.
- AWS CMK: Accidental deletion of databases & other compute services is enabled which protects these services against accidental deletion or termination.
Backup & Recovery Solution
We will have all production systems backed up daily for incremental data, weekly for full backups and 60 days retention. We use AWS native tools like snapshot & RDS backup features to achieve comprehensive backup strategy.
Our primary on-premise solution for backup is based on snapshot where in the data on EBS volumes is backed to Amazon S3 by taking point-in-time snapshots. Snapshots are incremental backups, which means that only the blocks on the device that have changed after most recent snapshot are saved. This minimizes the time required to create the snapshot and saves on storage costs.
For application code & other configurations we rely on AWS Code Commit repositories and CloudFormation templates for backing up. In addition to this, application code base is also stored in external Git Hub Repos for additional redundancy. Maintenance of these external Git Hub Repos is outside the scope of this project and currently being handled by the client.
|Critical Services||Backup Methodology|
|Simple Storage Service (S3)||As they get around 500 000 receipts a day and need to be sure to be able to recover some of these in case of emergency, we have enabled versioning on their S3 buckets and have a lifecycle policy to remove old receipts after 30 days due to GDPR regulations. To be able to restore receipts (objects) we are using AWS to be able to get the previous versions.|
|Aurora RDS||We have also enabled 30 days of backups on their database so they will be able to restore their receipts metadata and s3 locations. To be able to restore content from RDS we either spin up a new instance from snapshot to be able restore specific content.|
After careful considerations of all objectives and specific requirements, Opsio delivered the solution based on principles for DevSecOps.
|Design Principle||Solution Highlights|
|Achieve Faster Go to MarketFlexibility and Portability||Standard Provisioning/De-provisioning in a matter of minutesAutomated provisioning based on pre-defined conditions using CloudFormationFlexibility to dynamically port systems across various AWS regionsAbility to keep un-used systems in Dormant state and make them available only when required (e.g. DR Systems)Free from hardware upgrades|
|CAPEX to OPEXReduced Cost of Hosting & OperationsNo Lock DownPay as you Use||Pay-per-Use ModelNo lock down period; Switch On/Switch Off in an instantCombination of Reserved and On Demand instances providing best of both worlds – On Demand availability at Pre-defined rates No disruption to business due to Hardware related migrations|
|Innovative Service Delivery ModelOptimal Skills Provisioning||Unified approach for service deliveryCross-skilled teams for optimal costs|
|Optimal System Availability & Performance||Dynamic System Availability based on actual business requirement (e.g. DEV systems can operate for 8 hours/day to 24 hours/day depending upon development cycle)Enterprise Security on a Public cloud|
AWS Services Used
Here is a partial list of key AWS Services used in building customer’s platform.
- AWS CodePipeline: We are using AWS CodePipeline to fully managed continuous delivery service in ETNetwork platform. This helps us in automating our release pipelines for fast and reliable application and infrastructure updates. CodePipeline automates the build, test, and deploy phases of our release process every time there is a code change, based on the release model we have defined.
- AWS CodeBuild: AWS CodeBuild is used as a fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to be deployed. With CodeBuild, we don’t need to provision, manage, and scale our own build servers. CodeBuild scales continuously and processes multiple builds concurrently, so your builds are not left waiting in a queue.
- AWS CodeDeploy: AWS CodeDeploy is used to rapidly release new features, helps us to avoid downtime during application deployment, and handles the complexity of updating our applications.
- AWS CodeStar: We are using AWS CodeStar for notifying developers of Git Pull & Push requests. Also approval notifications are sent using CodeStar notification. By using AWS CodeStar, we have been able to set up our entire continuous delivery toolchain, allowing us to start releasing code faster.
- AWS CodeCommit: AWS CodeCommit is a secure, highly scalable, managed source control service that hosts private Git repositories. It makes it easy for our teams to securely collaborate on code with contributions encrypted in transit and at rest. CodeCommit eliminates the need for us to manage our own source control system or worry about scaling its infrastructure.
- AWS S3: We use Amazon Simple Storage Service (Amazon S3) as an object storage service that is used to host our static assets such as Images, documents, css & svg etc. This means these resources can be shared across a multitude of frontend systems communicating with our Backend.
- Amazon Elastic Container Service: Amazon Elastic Container Service (Amazon ECS) is a fully managed container orchestration service that help us to easily deploy, manage, and scale containerized applications for the customer. It deeply integrates with the rest of the AWS platform to provide a secure and easy-to-use solution for running container workloads in the cloud.
- AWS CloudFormation: AWS CloudFormation gives us an easy way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their life-cycles, by treating infrastructure as code. In the CloudFormation template we describe our desired resources and their dependencies so they can be launched and configured together as a stack.
- AWS Secrets Manager: AWS Secrets Manager helps us to protect secrets needed to access our applications, services, and IT resources. We use Secrets manager to easily store, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.
- AWS RDS: We are using AWS Aurora to set up, operate, and scale a relational database in the cloud storing all business critical data. It provides cost-efficient and resizable capacity while automating time-consuming administration tasks such as hardware provisioning, database setup, patching and backups.
- AWS SSM Parameter Store: Parameter Store, a capability of AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. We use it to store data such as passwords, database strings, DockerHub credentials and license codes as parameter values. We can store values as plain text or encrypted data.
- Amazon EC2 Systems Manager: AWS Systems Manager is the operations hub for AWS. We use it to access EC2 Service. Primarily for hopping to Jumpboxes without the need to setup additional SSH tools.
- AWS Backup: We use AWS to centralize and automate data protection of critical AWS services. It offers a cost-effective, fully managed, policy-based service that further simplifies data protection at scale.