Journey to a Successful AWS Deployment – Part 5 Cloud Cost Optimisation
Insight recently partnered with Amazon Web Services (AWS) to help customers with their cloud journey. Each customer’s requirements for the cloud can be different, from applications born in the cloud to a “lift and shift” migration of existing workloads to the cloud. A consultative approach is taken to ensure the migration to the cloud is efficient and fit for purpose.
What is common between each approach, is how the solution is architected to result in a successful deployment. Insight approach each design following the AWS Well-Architected Framework. The framework is a published set of best practices for deployment to AWS services. This framework provides consistency across deployments.
The AWS Well-Architected Framework consists of five pillars:
- Operational Excellence
- Performance Efficiency
- Cost Optimisation
The following post is part 5 of a blog series covering each of the five pillars and how Insight use these whilst architecting in the cloud. AWS provide a whitepaper on this subject – AWS Well-Architected Framework.
The fifth pillar in the AWS Well-Architected Framework is Cost Optimisation. This pillar covers how to apply best practices to cost optimisation in the cloud. In comparison to traditional data centre’s that had to be sized on predicted future usage and maximum capacity, cloud can provide a costing model in line with demand and optimised to minimise costs.
The design principles covered are as follows:
- Adopt a Consumption Model: Traditional on-premises data centre workloads had to be sized for maximum capacity and for future growth, this could prove costly with over provisioned hardware on different procurement cycles. Cloud resources can be charges as it is consumed and increase and decrease on demand. Workloads can be started up and shut down only when they are required using automation to only pay for when capacity is consumed.
- Measure Overall Efficiency: Measure the business output of the system with associated costs of delivering it. Use this data as a measure of gains when increasing output or reducing cost.
- Stop Spending Money On Data Centre Operations: Running an on-premises data centre can prove costly. AWS manage the underlying data centre so the customer can simply consume the services. This reduces overall Total Cost of Ownership (TCO).
- Analyse and Attribute Expenditure: Cloud resources can be easily measured against usage and cost, this allows for an easy way to attribute cost of applications to revenue streams and measure Return of Investment (ROI).
- Use Managed Services to Reduce Cost Of Ownership: Adopt AWS manage services where possible to reduce the operational overhead of underlying server infrastructure. Managed services can offer cloud scale and overall lower costs.
Cost Optimisation is broken down into four focus areas of best practice:
- Cost-Effective Resources
- Matching Supply with Demand
- Expenditure Awareness
- Optimising Over Time
One of the keys to cost saving is choosing the correct resource for the required workload. Insight can assist to help analyse and pick the correct architecture along with the correct resource types and configuration options.
Ensure resources are appropriately provisioned using assessments for existing application and workloads or correctly plan for new workloads. Managed services also need to be provisioned correctly so that access capacity is kept at a minimum. This can be done using Amazon CloudWatch to monitor usage. Services such as Amazon EMR or Amazon Redshift can scale out or in to meet demand but keep in mind the impact when adjusting capacity, time and effort, to scale certain services could negate the initial cost saving.
Managed services remove the burden required to maintain and operate the underlying infrastructure to instead focus on the business outcomes. Managed services can initially be more expensive but with this burden removed and the ability to operate at cloud scale can offer an overall lower cost. Examples of managed services are database services such as Amazon RDS and Amazon DynamoDB or Serverless services such as AWS Lambda and Amazon SQS.
Outside of manage services, services such as EC2 should be right sized using the lowest cost resource that still meets the technical requirement. The cost of the resources can be affected by not only demand but also AWS price drops with new generations launched.
Right sizing resources can again leverage Amazon CloudWatch for monitoring and gathering logs centrally, ensure monitoring reflects a time period that matches user activity to ensure performance is not affected.
Along with right sizing, its critical to choose the correct purchasing option for the required workload or application behaviour. AWS offer the following EC2 instance types.
- On-Demand: Pay a flat hourly rate with no long-term commitment. Good choice for applications or processes that will only be short term but that still require to be persistent.
- Spot Instances: Set a maximum bid price for instances. Instances will be available whilst the bid price is equal to or below the entered maximum price. Should the maximum price be exceeded, the instance will be stopped with a 2 minute warning. Great for batch processing or image processing where high compute is only needed for a short period of time. Spot instances can save up to 90% of on-demand pricing.
- Reserved Instances: Costs can be reduced by committing to using a certain amount of resources by using Reserved Instances. Instance types can be reserved for one or 3 years with up to 75% savings over on-demand. Reserved Instances are a great way to save costs for workloads that have a steady state or predictable usage. Reserved Instances are supported for a number of services such as EC2, Amazon RDS, Amazon Redshift and Amazon DynamoDB. Insight can assist with the correct Reserved Instance configuration, for instance, capacity can be reserved when using standard Reserve Instance within a specific AZ, pricing benefits can be shared between multiple accounts and instance types can be selected as convertible to change the family type as required.
Consider the region when selecting resources, each region must operate within the local market condition that can affect the pricing. By adopting services such as AWS CloudFormation or AWS CodeDeploy, applications can be deployed into different regions to evaluate costs as well as using the AWS Cost Calculator.
Matching Supply and Demand
One of the key drivers to adopt AWS is elasticity and scale, customers can consume virtually unlimited capacity of the cloud. By using Auto Scaling feature, compute can be scaled up and scaled down to meet demand. Scaling can be triggered by demand using Amazon CloudWatch that can monitor metrics such as CPU utilisation or network throughput.
If the business can predict busy periods, resource capacity can be aligned to meet demand using Auto Scaling. Instead of scaling on utilisation, scaling can be scheduled for a known time period. Resources can be scaled out and scaled back – consider scaling down outside of business hours or outside of known busy periods to save costs.
To reduce load on certain application types such as batch processing, a buffer can be used to reduce the demand if that write load doesn’t need to be processed immediately, reducing the load on the application. Amazon SQS can be used to decouple the application and provide queues for processing. The consumers can then receive the data in a fixed fashion, reducing the load and adds resiliency as Amazon SQS can be configured with a first in / first out (FIFO) queue to ensure the message is processed should an instance fail.
To optimise costing further and remove the fixed costs of running servers such as EC2 instances, AWS Lambda can provide a serverless approach to processing and further reducing costs.
To identify cost-reduction opportunities it’s important to understand the business. Businesses can often operate multiple systems ran by multiple teams. Make sure the relevant stakeholders are involved when considering cost optimisation for the cloud journey. Financial controllers should be involved and understand the cloud model of consumptions, business unit owners must understand the cloud model to provide direction, and technical leads must understand the impact their choice of resources has on costs.
Cost optimisation begins with a granular understanding of the breakdown in spend across the business and its use in the cloud. Native AWS billing tools can be leveraged to estimate and forecast costs based on usage, receive notifications when spend will exceed a budget and report on Reserve Instance types. Insight can also assist with this process beyond the native tools and can offer a standard billing platform.
Consider account structuring and tagging to drive cost management to systems, revenue streams, business units and products. Account structuring can be implemented in a many to one fashion, for instance a root parent account with children accounts for business units and teams. This can be beneficial for administrative tasks, billing isolation between business units and setting AWS service limits to specific workloads.
Using account structuring, consolidated billing can be implemented. Consolidated billing uses a single payer account with linked accounts assigned to business units and teams. Visibility into each linked account is still possible and volume discount on services such as Amazon S3 can be aggregated across accounts as well as Reserved Instance discounts and benefits across the accounts.
Apply tags to resources to help categorise and track costs to the relevant business information. Apply tags that can represent the business unit, application name, service team or relevant project. Tags help to manage costs and help govern resources across the environment.
Optimising Over Time
Monitor and measure over time to ensure implemented systems are cost effective for the business. Consider implementing a Cloud Centre of Excellence to co-ordinate all aspects of cost optimisation from technical systems to people or processes.
Continually monitor ongoing cost that can be measured against set goals, for instance aim to reduce cost per transaction or output over a set period of time, monitor and measure elasticity usage and continually evaluate against available purchasing options.
Keep an eye on new AWS releases and features that could be adopted for the organisation, new features could replace old ones that provide further efficiency and reduce costs.
Cost optimisation is the fifth and final pillar within the AWS Well-Architected Framework and with all of the framework, it represents an ongoing effort. By planning the AWS deployment up front and following this methodology, it ensures best practice is followed from the start and not bolted on as an afterthought.
If you are interested in finding out more, please contact your Insight Account Manager or get in touch via our contact form here.
Why not also read ‘Journey to a Successful AWS Deployment – Part 1 Cloud Operations’?