AWS Ephemeral Environments: Tutorial With Example

Modern applications have code releases daily or weekly, requiring developers and test engineers to respond accordingly. Ephemeral environments—temporary and highly dynamic setups hosted on cloud platforms like Amazon Web Services (AWS)—have emerged as a pivotal enabler for supporting frequent software releases. These environments allow teams to mirror production settings for testing, staging, and deployment cost-effectively and flexibly to streamline tasks like end-to-end integration testing.

This article delves into the multifaceted benefits of ephemeral environments, exploring how AWS supports these transient setups. It outlines the architectural foundations necessary for creating and managing ephemeral environments, discusses their advantages, addresses the inherent challenges, and presents practical steps for implementing them using AWS services.

Summary of key AWS ephemeral environments concepts

The table below summarizes the core concepts covered in this article.

The benefits of hosting ephemeral environments on AWS

Ephemeral environments, such as preview environments, offer substantial benefits for software development by providing isolated, temporary testing and staging setups that mirror production environments. These environments eliminate many bottlenecks associated with traditional staging processes and facilitate a more agile development lifecycle. Leveraging Amazon Web Services (AWS) can further enhance these benefits, introducing greater scalability, flexibility, and cost-effectiveness to the ephemeral environment strategy.

Here are some specific areas where ephemeral environments provide advantages:

• AWS integration for scaling and flexibility: AWS provides services that can be seamlessly integrated into managing ephemeral environments, making them more dynamic and scalable. AWS services like AWS Elastic Kubernetes Service (EKS) and AWS Fargate enable container management without requiring physical server maintenance, aligning well with the temporary nature of ephemeral environments.
• Auto-scaling and load balancing: AWS Auto Scaling ensures that the ephemeral environments have the right computing resources based on the workload. This is particularly useful when there are uneven workloads, such as end-stage testing phases or when multiple developers push changes simultaneously. AWS Elastic Load Balancing can also distribute incoming traffic across Amazon EC2 instances, containers, and IP addresses, improving applications' performance and fault tolerance.
• Enhanced testing and deployment: Using AWS CodeBuild and AWS CodeDeploy, ephemeral environments can benefit from automated building and deployment processes. This integration ensures that environments are always up-to-date with the latest builds, which is crucial for testing in a state that closely mimics production.
• Automated rollbacks and canary deployments: AWS CodeDeploy also supports automated rollbacks and canary deployments. If an issue is detected during deployment to an ephemeral environment, AWS CodeDeploy can automatically roll back to the last known good state, significantly reducing downtime and disruption. Canary deployments allow the incremental introduction of changes, which can be tested in ephemeral environments to ensure stability before full-scale deployment.
• Security and compliance: AWS’s security model helps address compliance controls, activity monitoring, and automatic security assessments with AWS Inspector. This ensures that even temporary environments adhere to strict security standards.
• Identity and access management (IAM): AWS identity and access management enables granular control over who can access specific resources within an ephemeral environment. This is crucial for teams that need to ensure secure and compartmentalized access to different parts of their applications during various stages of the development process.
• Cost management and optimization: AWS offers a pay-as-you-go model that aligns well with the ephemeral nature of preview environments. This model allows organizations to pay only for their computing usage, which can significantly lower costs compared to maintaining permanent resources.
• Using Amazon EC2 Spot Instances: To further reduce costs, teams can utilize Amazon EC2 Spot Instances for running ephemeral environments. Spot Instances allow users to use unused AWS EC2 capacity at a significant discount. Since ephemeral environments are inherently temporary and can tolerate interruptions, they are ideal for Spot Instances.

{{banner-large-dark="/banners"}}

Building blocks for ephemeral environments: Core AWS services

Amazon EC2 and EC2 spot instances

The Amazon Elastic Compute Cloud instances are the most common service for hosting ephemeral AWS environments. The EC2 service makes it possible to launch virtual servers as needed. As mentioned above, EC2 Spot Instances are ideal for saving money in non-critical environments that can withstand interruptions.

AWS Elastic Kubernetes Service (EKS) and AWS Fargate

For containerized applications, AWS Elastic Kubernetes Service (EKS) offers a managed Kubernetes service that simplifies running Kubernetes on AWS without installing and operating your own Kubernetes control plane. This is particularly useful for ephemeral environments where the infrastructure needs to be as agile as the development process. AWS Fargate further enhances this by providing a serverless compute engine for containers, removing the need to manage servers or clusters and allowing developers to focus on building applications.

Karpenter, an open-source Kubernetes autoscaler, can be integrated with EKS and Fargate for optimal resource management. Karpenter intelligently scales nodes up and down based on pod requirements, ensuring your ephemeral workloads always have the right resources and minimizing costs.

AWS Elastic Container Service (ECS)

Elastic Container Service (ECS) is a highly scalable and performant container orchestration service. It simplifies the process of deploying, managing, and scaling containerized applications on AWS. ECS is well-suited to ephemeral environments because it allows you to easily create and tear down containers as needed, optimizing resource utilization and adapting to changing workloads It’s a practical alternative for users who aren’t yet ready to adopt Kubernetes but have containerized workloads.

AWS Lambda

AWS Lambda is well suited for event-driven, ephemeral workloads. It allows you to run code that responds to triggers, such as data or system state changes, without provisioning or managing servers. This functionality aligns perfectly with the principles of ephemeral environments by running only when needed and scaling automatically with the workload.

Building blocks for ephemeral environments: Automation and orchestration

In the realm of ephemeral environments, automation and orchestration are fundamental building blocks for achieving scalable and flexible infrastructure management.

AWS CloudFormation

AWS CloudFormation allows the creation of templates that can be used to deploy and manage AWS resources consistently and predictably to manage and provision resources as code. This is especially useful in ephemeral environments where infrastructure needs to be replicated or destroyed systematically based on the lifecycle of development branches.

AWS CodePipeline and AWS CodeBuild

Automation is key to managing ephemeral environments efficiently. AWS CodePipeline provides a continuous integration and continuous delivery (CI/CD) service for fast and reliable application and infrastructure updates.

CodePipeline automates your release process’s build, test, and deploy phases based on the release model you define every time a code change occurs. AWS CodeBuild is a fully managed service that compiles source code, runs tests, and produces software packages that are ready to deploy and can be integrated into CodePipeline workflows for automated processing.

Terraform and Pulumi

Tools like Terraform and Pulumi stand out by offering robust Infrastructure as Code (IaC) capabilities, enabling developers to define and manage infrastructure through code programmatically. Terraform uses a declarative approach, allowing users to specify their desired end-state, while Pulumi supports multiple programming languages, providing a more familiar syntax for software developers. Both platforms streamline the deployment and maintenance of ephemeral environments, enhancing reproducibility and reducing manual overhead, thus facilitating rapid provisioning and de-provisioning of resources in response to changing development and operational needs.

{{banner-small-4="/banners"}}

Building blocks for ephemeral environments: Security and compliance

AWS identity and access management (IAM)

Security is non-negotiable, even in temporary environments. AWS IAM helps manage access to AWS services and resources securely. Using IAM, you can define who is authorized to access which resources, which is crucial for protecting access to the AWS resources used in ephemeral environments. You can define identities (users, applications, etc.) and specify what actions they can perform on specific AWS services. This ensures that users only have the necessary access to your resources.

Amazon Virtual Private Cloud (VPC)

Amazon VPC lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a defined virtual network. This control over the virtual networking environment—including IP address ranges, the creation of subnets, and the configuration of route tables and network gateways—is essential for creating secure and isolated ephemeral environments.

AWS Secrets Manager

When managing infrastructure, it is crucial to maintain distinct credentials for production and non-production environments to minimize security risks. Distributing access too broadly, particularly in environments where many developers have access to Continuous Integration (CI) systems, can be perilous. Overly permissive CI access could lead to accidental or malicious changes that compromise security. To enhance security measures, integrating AWS Secrets Manager is advisable. This tool securely manages, retrieves, and stores secrets, ensuring that sensitive information such as API keys, passwords, and credentials are handled with the utmost security, reducing the risk of exposure and bolstering overall system integrity.

An example of an AWS ephemeral environment with step-by-step instructions

This section reviews a simple example of a Python-based Flask app and provides instructions for creating an ephemeral environment using GitHub Actions, AWS EKS, and AWS ECR.

The requirements are as follows:

• Whenever a pull request (PR) is created, an ephemeral environment should be created.
• A Kubernetes deployment gets executed that contains functional tests.
• Once the deployment is finished, the environment is destroyed.

Step 1: Set up your AWS ECR repository

First, create a repository in ECR to store your Docker images.

Create the ECR repository:

aws ecr create-repository --repository-name ephemeral-envs/myapp

Next, authenticate Docker or Podman to your ECR repository:

aws ecr get-login-password --region your-region | docker login --username AWS --password-stdin your-account-id.dkr.ecr.your-region.amazonaws.com

Build your Docker image, assuming you have a Dockerfile in your project directory:

docker build -t ephemeral-envs/myapp .

Now tag your Docker image for ECR:

docker tag ephemeral-envs/myapp:latest your-account-id.dkr.ecr.your-region.amazonaws.com/ephemeral-envs/myapp:latest

Finally, push your Docker image to ECR:

docker push your-account-id.dkr.ecr.your-region.amazonaws.com/ephemeral-envs/myapp:latest

Step 2: Set up your EKS cluster

If you haven’t done so already, set up an EKS cluster. You can do this using the AWS Management Console, AWS CLI, or eksctl.

Here’s a quick way to do this with eksctl:

eksctl create cluster --name my-ephemeral-cluster --nodes 2

Step 3: Update kubeconfig

To obtain access to a new cluster, you would need to run the next command with aws cli:

aws eks --region ${{AWS_REGION}} update-kubeconfig --name your-eks-cluster-name

{{banner-small-1="/banners"}}

Step 4: Deploy your application to EKS

Create a Kubernetes Deployment YAML File called myapp-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: your-account-id.dkr.ecr.your-region.amazonaws.com/ephemeral-envs/myapp:latest
        ports:
        - containerPort: 80

Now, optionally, create a Kubernetes service YAML file.

If your application needs to be accessible, create a service called myapp-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
  labels:
    app: myapp
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: myapp

Finally, apply your deployment and service to your EKS cluster:

kubectl apply -f myapp-deployment.yaml
kubectl apply -f myapp-service.yaml

Step 5: GitHub Actions workflow for ephemeral environments

You’ll need two main workflows: creating/updating the ephemeral environment when a pull request is opened or updated and destroying the environment when the pull request is closed.

First, be sure to do the following:

• Ensure that your GitHub repository has secrets configured for AWS access:some text
  • AWS_ACCESS_KEY_ID: Your AWS access key.
  • AWS_SECRET_ACCESS_KEY: Your AWS secret access key.
  • AWS_REGION: Your AWS region, e.g., us-east-1.
• Be sure that your EKS cluster is already set up and accessible with kubectl.
• Check that you have a Dockerfile and Kubernetes manifest files (deployment.yaml and optionally service.yaml) in your repository.

To create or update an ephemeral environment, create a file in your repository under the .github/workflows/ directory, such as .github/workflows/deploy-ephemeral-env.yml:

name: Deploy Ephemeral Environment

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ secrets.AWS_REGION }}

    - name: Build Docker image
      run: |
        docker build -t myapp:${{ github.event.pull_request.head.sha }} .
        
    - name: Authenticate Docker to Amazon ECR
      run: aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin your-account-id.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
    
    - name: Tag and Push Docker image to Amazon ECR
      run: |
        docker tag myapp:${{ github.event.pull_request.head.sha }} your-account-id.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/myapp:${{ github.event.pull_request.head.sha }}
        docker push your-account-id.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/myapp:${{ github.event.pull_request.head.sha }}
    
    - name: Set up Kubeconfig
      run: aws eks --region ${{ secrets.AWS_REGION }} update-kubeconfig --name your-eks-cluster-name

    - name: Deploy to EKS
      run: |
        kubectl set image deployment/myapp-deployment myapp=your-account-id.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/myapp:${{ github.event.pull_request.head.sha }}
        # Optionally, apply your service and any other K8s resources here

To destroy an ephemeral environment, create another file under the .github/workflows/ directory, such as .github/workflows/destroy-ephemeral-env.yml:

name: Destroy Ephemeral Environment

on:
  pull_request:
    types: [closed]

jobs:
  teardown:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ secrets.AWS_REGION }}

    - name: Set up Kubeconfig
      run: aws eks --region ${{ secrets.AWS_REGION }} update-kubeconfig --name your-eks-cluster-name

    - name: Remove from EKS
      run: |
        kubectl delete deployment myapp-deployment
        # Optionally, delete your service and any other K8s resources here

Notes:

• Replace placeholder values (e.g., your-account-id and your-eks-cluster-name) with your actual AWS account ID and EKS cluster name.
• These workflows assume that your application is packaged as a container and deployed as a Kubernetes deployment in EKS.
• The deploy-ephemeral-env.yml workflow builds a Docker image tagged with the PR’s commit SHA, pushes it to ECR, and updates the Kubernetes deployment to use this image. This ensures that each PR has an isolated environment.
• The destroy-ephemeral-env.yml workflow triggers when a PR is closed, cleaning up the deployed resources

{{banner-small-2="/banners"}}

Challenges with creating an ephemeral environment on AWS

Ephemeral environments are ideal for use with continuous integration, deployment, and testing (CI/CD) pipelines hosted on AWS. However, setting up and managing these environments presents unique challenges that organizations must navigate:

• Automation complexity: Automating the setup and teardown of ephemeral environments is critical but challenging, involving scripting or using infrastructure as code (IaC) tools like AWS CloudFormation or Terraform. Developing these automation scripts requires deep knowledge of AWS services and IaC principles, which can represent a steep learning curve for teams.
• Configuration management: Each ephemeral environment should ideally be a replica of the production setting to ensure testing accuracy. Checking that configurations are consistent across all these environments while keeping them isolated from production is complex and error-prone. Configuration drift can occur if configurations are not meticulously managed, leading to inconsistencies and failures in testing.
• Resource management and cost control: Although ephemeral environments are temporary, they can rapidly consume significant cloud resources if not managed properly. Effective scaling and termination policies must be in place to avoid unnecessary costs. AWS offers services like AWS Auto Scaling and AWS Budgets to help manage these aspects, but setting them up requires careful planning and monitoring.
• Security concerns: There must be a balance between accessibility for testing and protection against security breaches. Implementing adequate security measures in a dynamic and temporary environment—such as identity and access management, data encryption, and proper network configurations—complicates security management.
• Integration and dependencies: Ephemeral environments often need to integrate with existing systems and services, which may have complex dependencies. Managing these integrations and ensuring that all systems communicate effectively without disruptions can be daunting, especially when environments are continuously created and destroyed.
• Performance metrics and monitoring: Like persistent environments, ephemeral environments must be monitored for performance and capacity. Tools like AWS CloudWatch can be utilized for monitoring, but configuring these tools for short-lived environments that may change frequently involves overhead.

Key features and benefits of using Coherence

Coherence streamlines app development on AWS (or GCP). It creates complete development environments with automated build pipelines (including testing) from a simple YAML file. Key features include:

• Isolated environments for every code branch review
• Automated testing throughout the build pipeline
• Easy production deployment with a user-friendly interface
• Secure access to environments with built-in controls
• Simplified management of secrets and sensitive data

The following table describes various aspects and advantages of Coherence’s solutions in more detail.

{{banner-small-1="/banners"}}

Last thoughts on AWS ephemeral environments

Ephemeral environments in AWS represent a significant advance toward more agile, secure, and cost-effective software development practices. By leveraging AWS’s robust ecosystem, organizations can enjoy scalability, enhanced security, and decreased overhead costs, making these environments ideal for short-lived but critical development tasks.

However, the complexity of setting up and managing these environments cannot be overlooked. Addressing it requires a deep understanding of AWS services, meticulous planning, and strategic resource management. As we have explored, the judicious use of tools such as AWS CloudFormation, IAM, and CodePipeline is essential. By adhering to best practices for automation, security, and cost management, developers can harness the full potential of ephemeral environments to accelerate development cycles and improve product quality.