Select Page
A PaaSing Trend: Why enterprises will finally adopt PaaS in 2015

A PaaSing Trend: Why enterprises will finally adopt PaaS in 2015

platformIf there is one universal truth in  IT, it is that businesses are always slower to adopt new technology than expected.  Way back in March of 2011, Gartner declared that it would be the year of Platform as a Service (PaaS), predicting that by 2015 most enterprises would have part of their critical software leveraging PaaS directly or indirectly.  More recently in January of 2014, 451 Research published a report predicting that lackluster PaaS adoption may be an indication that standalone PaaS services may simply be consolidated into existing Infrastructure as a Service (IaaS) offerings.

For most businesses I meet, accelerating the application development life-cycle, and simplifying all operations processes remains a key challenge.  In fact, the majority of customers I have spoken with in the last six months since 451’s research either have active projects to develop their own internal PaaS offering, or intend to start such projects within the year.  It may be small sample bias, but talk to enough businesses and users, and I think that patterns begin to emerge.  In summary, I believe we are seeing what may be the very beginning of a larger shift (finally) toward enterprise IT developing PaaS-like capabilities of their own.

Yet in the end, 451’s prediction may not be too far from the truth, as new technologies enable existing IaaS solutions to become more PaaS-like.  I have written previously on the adoption of containerization and the resulting consequences for PaaS and IaaS.  Since then, the activity in the Docker ecosystem has continued to accelerate.  Google announced Kubernetes for running containers on the Google Cloud Platform, Amazon announced support for Docker containers in AWS Elastic Beanstalk, while Red Hat and Ubuntu both announced support for Docker.  The takeaway then is that vendors are moving very quickly to enhance their products and services with native support for containers, and as a result PaaS-like features will start to emerge as a standard part of IaaS offerings.

Then there is the issue of operations. For  years now, people like me have been drawing charts like the one below, which identified the trend toward platform and utility computing. The utopian theory goes that application developers want to focus strictly on creating their core business logic, and leave the messy work of hosting and scaling the application to… no one at all.  In the future, ever increasing layers of automation and abstraction eliminate the necessity of operations (and operations teams) completely, ultimately leading to the unicorn-infested utopia of NoOps.

it-evolution

The problem with this chart, and with this trend, has always been this NoOps aspect.  What we saw recently with the Great Code Spaces Debacle of 2014, and as Trevor Pott described so succinctly, is what most IT operations and sysadmins understand intuitively: cloud services are not a magic pill to eliminate operations issues.  Instead, they create a number of new challenges for operations, such as data replication and migration, disaster recovery, security, and governance, that can all have dire consequences when ignored.  These challenges are significant barriers to wholesale PaaS adoption, and some of the issues Trevor Pott identifies (like lock-in) are reasons PaaS adoption has been stalled versus the earlier rose-colored forecasts.

Mitigating these risks requires more traditional systems management techniques, yet those techniques are precisely what has been driving developers toward PaaS in the first place.   Containerization, and its emerging ecosystem, will result in a new set of tools that lead to “open” PaaS architectures.  Just like the autopilot in a commercial aircraft, in the “Open PaaS” world, operations teams will be able to disengage the automation to analyze, troubleshoot, and correct infrastructure issues in real time.   This means that PaaS solutions like OpenShift and Cloud Foundry, or their successors, will have to do a lot more to create tools that offer administrators command, control, and logging.  OpenShift is a step in the right direction, and SaltStack already provides a lot of this tooling right out of the box.  When PaaS solutions start to look a lot more like open architectures, we will see enterprises moving quickly to enable their development teams, creating PaaS-like features for developers without PaaS-like lock-in.

 

Automating application deployments across clouds with Salt and Docker

Automating application deployments across clouds with Salt and Docker

saltstack-logoIf you have not had the chance to work with Salt yet, it is a very exciting new configuration management system that is easy to get up and running, powerful enough to support distributed command execution and complex configuration management, and scalable enough to support thousands of servers simultaneously.

Recently, I wrote about how a de facto containerization standard will enable a whole new generation of management tools.  Back in January, SaltStack announced several awesome new features in the Salt 2014.1.0 (Hydrogen) release, including support for the life-cycle management of Docker containers.  SaltStack is very early in getting Docker support, and Docker itself still does not consider itself production-ready (tell that to Yelp and Spotify), but together these tools offer an out-of-the box solution for getting started with immutable infrastructure.

The deployment and management of an application across multiple virtual machines, using multiple public clouds, is a use case that would have been considered categorically “hard” just a year ago.  Companies like Google, Facebook, and Ning spent many years developing this kind of orchestration technology internally in order to deal with their scale challenges. Today, using Docker containers, together with Salt-Cloud, and a few Salt States, we can do this from scratch in a few tens-of-minutes of effort.  And, because we are using Salt’s declarative configuration management, we can scale this pattern to actually operate our production environment.

Use Case

salt-docker-use-cases

The core use case is one or more application containers which we want to deploy on one or more virtual machines, using one or more public cloud providers.

For the sake of simplicity, we will restrict this use case to:

  • Assume some familiarity with Salt
  • Assume some familiarity with Docker
  • Assume that you have a Salt master already installed
  • Assume that you want to do this on a single public cloud, using Digital Ocean (since adding new clouds to Salt-Cloud is dead simple)
  • Simulate a real-world application with a dummy apache service

Demo Use Case

docker-salt-digitalocean

In order to simulate a real-world application, will create a Docker container with the Apache web server.  Conceptually, this container could be a front-end proxy, a middle tier web service, a database, or virtually any other type of service we might need to deploy in our production application. To do that, we simply create a Dockerfile in a directory in the normal way, build the container, and push the container to the Docker repository.

Step 1: Create a Dockerfile

[email protected]:/some/dir/apache# cat Dockerfile
# A basic apache server. To use either add or bind mount content under /var/www
FROM ubuntu:12.04

MAINTAINER Kimbro Staken version: 0.1

RUN apt-get update && apt-get install -y apache2 && apt-get clean && rm -rf /var/lib/apt/lists/*

ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2

EXPOSE 80

CMD ["/usr/sbin/apache2", "-D", "FOREGROUND"]

Step 2: Build the container

[email protected]:/some/dir/apache# docker build -t jthomason/apache .
Uploading context  2.56 kB
Uploading context
Step 0 : FROM ubuntu:12.04
 ---> 1edb91fcb5b5
Step 1 : MAINTAINER Kimbro Staken version: 0.1
 ---> Using cache
 ---> 534b8974c22c
Step 2 : RUN apt-get update && apt-get install -y apache2 && apt-get clean && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 7d24f67a5573
...
<A WHOLE LOT OF STUFF HAPPENS>
...
Successfully built 527ad6962e09

Step 3: Push the container

[email protected]:/tmp/apache# docker push jthomason/apache
The push refers to a repository [jthomason/apache] (len: 1)
...
<A WHOLE LOT OF STUFF HAPPENS>
...
Pushing tag for rev [527ad6962e09] on {https://registry-1.docker.io/v1/repositories/jthomason/apache/tags/latest}

With our demo application successfully pushed to the Docker registry, we are now ready to proceed with orchestrating its deployment.  As mentioned previously, we assume you have a Salt master installed somewhere. If not, you’ll need to follow the documentation to get a Salt master installed.  The next step then is to configure Salt-Cloud for your choice of public cloud provider.  Configuring Salt-Cloud is simple.  We need to create an SSH key pair that Salt-Cloud will use to install the Salt Minion on newly created VMs, add that keypair to our choice of public cloud provider, and create a Salt-Cloud configuration file with the API credentials for our public cloud.

Step 4: Create an SSH Key Pair

[email protected]:/etc/salt/keys# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): digital-ocean-salt-cloud
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_dsa.
Your public key has been saved in digital-ocean-salt-cloud.pub.
The key fingerprint is:
06:8f:6f:e1:97:5a:5a:48:ce:09:f3:b6:33:42:48:9a [email protected]
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|                 |
|      .          |
|    .  +         |
|   + .+ S        |
|  E . [email protected] + .     |
|     .  @ =      |
|      .ooB       |
|       .+o       |
+-----------------+
[email protected]:/etc/salt/keys#

Step 5: Upload SSH Key Pair

digital-ocean-add-ssh-key

With Digital Ocean now enabled with our SSH keys, next steps before provisioning time are to configure salt-cloud with Digital Ocean’s API key for our account, and define the profiles for virtual machine sizes, geographic locations, and images for Digital Ocean.  The salt-cloud configuration for credentials are kept on the salt-master in /etc/salt/cloud.providers.d/, while the profiles for each public cloud are kept in /etc/salt/cloud.profiles.d/.  See the salt-cloud documentation for more details on configuration options.

Step 6: Configure Salt-Cloud

do:
  provider: digital_ocean
  # Digital Ocean account keys
  client_key: <YOUR KEY HERE>
  api_key: <YOUR API KEY HERE>
  ssh_key_name: digital-ocean-salt-cloud.pub
  # Directory & file name on your Salt master
  ssh_key_file: /etc/salt/keys/digital-ocean-salt-cloud
# Official distro images available for Arch, CentOS, Debian, Fedora, Ubuntu

ubuntu_512MB_sf1:
  provider: do
  image: Ubuntu 12.04.4 x64
  size: 512MB
#  script: Optional Deploy Script Argument
  location: San Francisco 1
  script: curl-bootstrap-git
  private_networking: True

ubuntu_1GB_ny2:
  provider: do
  image: Ubuntu 12.04.4 x64
  size: 1GB
#  script: Optional Deploy Script Argument
  location: New York 2
  script: curl-bootstrap-git
  private_networking: True

ubuntu_2GB_ny2:
  provider: do
  image: Ubuntu 12.04.4 x64
  size: 2GB
#  script: Optional Deploy Script Argument
  location: New York 2
  script: curl-bootstrap-git
  private_networking: True

It is now time to configure Salt to provision our application container.  To do that, we need to create two Salt states, one to provision Docker on a newly created VM, and another to provision the application container.  Salt states are Salt’s declarative configuration states, which are executed on target hosts by the salt-minion.  States are an incredibly rich feature of Salt, one that could hardly be covered with any sufficient level of detail in a tutorial like this.  This example is not a particularly smart or optimal use of states, but it is simple. You’ll want to read up on Salt states to develop the best practices for your environment.

Step 7: Create a Salt state for Docker

docker-python-apt:
  pkg.installed:
    - name: python-apt

docker-python-pip:
  pkg.installed:
    - name: python-pip

docker-python-dockerpy:
  pip.installed:
    - name: docker-py
    - repo: git+https://github.com/dotcloud/docker-py.git
    - require:
      - pkg: docker-python-pip

docker-dependencies:
   pkg.installed:
    - pkgs:
      - iptables
      - ca-certificates
      - lxc

docker_repo:
    pkgrepo.managed:
      - repo: 'deb http://get.docker.io/ubuntu docker main'
      - file: '/etc/apt/sources.list.d/docker.list'
      - key_url: salt://docker/docker.pgp
      - require_in:
          - pkg: lxc-docker
      - require:
        - pkg: docker-python-apt
      - require:
        - pkg: docker-python-pip

lxc-docker:
  pkg.latest:
    - require:
      - pkg: docker-dependencies

docker:
  service.running

This first salt state defines the dependencies and configuration for installing Docker on

Step 8: Create Salt state for the application container

apache-image:
   docker.pulled:
     - name: jthomason/apache
     - require_in: apache-container

apache-container:
   docker.installed:
     - name: apache
     - hostname: apache
     - image: jthomason/apache
     - require_in: apache

apache:
   docker.running:
     - container: apache
     - port_bindings:
            "80/tcp":
                HostIp: ""
                HostPort: "80"

Now that configuration is complete, we are now ready to provision 1..n virtual machines, each with a running instance of our application container.  Before we do that, let us first verify that the Salt master is actually working. We know there is at least one agent that should be talking to this salt-master at this point, which is the agent running on the salt-master itself.

Step 9: Verify that Salt is working

[email protected]:~# salt '*' test.ping
salt.garply.org:
True
[email protected]:~#

Satisfied that everything is in working order with the salt installation, we can now provision our first virtual machine with an instance of our container using salt-cloud.

Step 10: Provision a VM with an instance of the container

[email protected]:# salt-cloud --profile ubuntu_512MB_sf1 one.garply.org
[INFO    ] salt-cloud starting
[INFO    ] Creating Cloud VM one.garply.org
[INFO    ] Rendering deploy script: /usr/lib/python2.7/dist-packages/salt/cloud/deploy/curl-bootstrap-git.sh
...
<A WHOLE LOT OF STUFF HAPPENS>
...
one.garply.org:
    ----------
    backups_active:
        False
    created_at:
        2014-04-23T19:23:12Z
    droplet:
        ----------
        event_id:
            22373933
        id:
            1521385
        image_id:
            3101045
        name:
            one.garply.org
        size_id:
            66
    id:
        1521385
    image_id:
        3101045
    ip_address:
        107.170.230.112
    locked:
        True
    name:
        one.garply.org
    private_ip_address:
        None
    region_id:
        3
    size_id:
        66
    status:
        new

After the salt-cloud run completes, it emits a YAML blob containing information about the newly created VM instance.  Let’s use the IP address of the instance to see if our application is running

Step 11: Verify application is running 

it-works

Great success!

We have established the basic setup and management pattern for our infrastructure.  Adding additional public clouds is easy, thanks to salt-cloud, providing a single control interface for our entire application infrastructure.   But where to go from here?    A starting point is to consider how salt states can be used to manage VM and container life-cycles, in the context of the overall continuous integration and deployment process.   I plan to share some of my thoughts on that specifically in a future post.  Obviously, there is a lot of thought to be given to your specific objectives, since that will ultimately determine the deployment and operations architecture for your application.  However, Salt is an incredibly powerful tool that, when combined with Docker, provides a declarative framework for managing the application life-cycle in the immutable infrastructure paradigm right out of the box.  That versatility puts a whole lot of miles behind you, allowing you to focus on other core challenges with application deployment and operations.

 

Why Docker and Containerization is a key enabling technology for PaaS

Why Docker and Containerization is a key enabling technology for PaaS

docker-logoLast week at the Red Hat Summit in San Francisco, CEO Jim Whitehurst called out containerization during his keynote as a stunning new development in open source, declaring that last year he “didn’t even know what Docker was”.  Mr. Whitehurst was in good company no doubt, since nothing like Docker’s meteoric rise has been seen in open source in quite some time.

Building on it’s previously announced support for Docker in RHEL and OpenShift, Red Hat reaffirmed it’s commitment to Docker, announcing Project Atomic, an innovative new deployment management solution leveraging Docker containers (and competing with Andressen-Horowitz startup CoreOS).  RedHat also announced geared, a command-line client and agent for linking Docker containers with systemd for OpenShift Origin.

Technology purists will be quick to point out that containerization is nothing new when it comes to virtualization.  Indeed, OpenVZ, the predecessor to Linux Containers (LXC) has been in wide use since 2005.  And Solaris, along with BSD derivatives, have supported their own containerization implementations for many years. But Docker’s popularity has less to do with technological advancement, and more to do with a fundamental shift in the operations model for IT. Docker’s tools for container management are arguably much more developer-friendly than previous containerization solutions, and those benefits could hardly come at a better time.

Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) are both infrastructure paradigms that each promised to reduce the operational complexity of IT infrastructure.  They did the opposite.  Ask your neighborhood CIO or CTO, and you will find out that software developers are spending increasing amounts of their time managing software deployments instead of writing software.   Add to that the increasing frequency of change driven by Agile development processes, and the new technology complexity introduced by IaaS and PaaS, and you have identified the pain that will (finally) drive the adoption of containers as a de-facto means of deploying software.

devops-container

The adoption of a de-facto containerization standard will have widespread implications, and many positive impacts. Containerization has always had some obvious technical advantages, including better resource utilization, and control of guest operating systems.  One of more nuanced benefits however has to do with abstraction of the operating system and its resource dependencies, from the application and its dependencies. This latter benefit helps to enable an idealized workflow called immutable infrastructure, where the state of the application configuration and dependencies is preserved from development, through testing, and on into production. Conceptually, immutable infrastructure is accomplished through deployment automation using introspection of the container itself, or through more traditional monolithic orchestration templates. Either way, the  difference is that containers provide a clean separation of concerns between development and operations dependencies. In summary, changes are no longer made to production, changes are made to containers, and containers have a finite life-cycle that is optimized for developer productivity and operational simplicity.

container-workfow

A whole new generation of platforms is already emerging.  Flynn.io is a new PaaS based on Docker that promises better productivity for developers, hackability, and optimized DevOps.  Mitchell Hashimoto, the creator of Vagrant, has been promoting the immutable infrastructure concept for some time and is now developing Serf, a service discovery and orchestration solution that can be used with Docker.   Finally, the aforementioned CoreOS has developed a lightweight Linux distribution that is Just Enough Operating System to manage Docker deployments.  These solutions each tackle a touch unsolved problem, container orchestration and the creation of container-based services, albeit with very different approaches.