Understanding Innovation, Creativity and Mastery

In recent years I've often heard the words "innovation" and "creativity" used interchangeably. Sometimes mastery and creativity are lumped together as well. But as a technologist, artist and agile coach I believe there is value in differentiating these terms in order to foster each of them. Why? Because a creation can be beautiful, "one of a kind" and highly customized without being innovative (and vice versa) and as such still has great value.

To illustrate the difference I will use realist oil painting, another genre I'm familiar with.

As a practiced artist, I have the tools and skills to paint oil portraits that are completely customized.  What makes them creative is that they are designed and crafted as a one of a kind object. By capturing a person's likeness in paint by hand, a conservator can not only distinguish between my work and that of other artists but they can also distinguish between two paintings by me of the same subject - because they are new and "one-of-a-kind".

 Merriam Webster's definition of Creativity; "The ability to make new things or think of new ideas"

Those painted images are creative because they were made as new things.  That particular person has never been painted before and their image has not been previously captured in that way. But I can say that creativity doesn't necessarily make all of my custom images "innovative." It is my great hope that they are beautiful and valuable so they will become a legacy for future generations. But even if a painter has "mastered" a technique from the past, that established, time tested technique is not innovative - but it can be used to paint a creative work of art. Let's also appreciate the value of "mastery" both with or without creativity. Those same techniques can be used by a skilled painter to create a work of art that is not 'creative' by the dictionary definition but it is still beautiful and masterful. That masterful work is valuable because of the time and skill required to make it.

Now let's consider how creativity and mastery can be combined in the work of innovation. Imagine that same work of art but this time the painter who has mastered a given technique goes beyond earlier techniques and does something groundbreaking and completely new. That painter may use an old technique and combine it with newly discovered pigment or substance that creates an effect nobody has ever seen before. The effect might be dramatic or so subtle that only the innovator knows why it works. ( Yes, this is still possible and happening today, I've seen other artists truly innovate and I've been told by a manufacturer that I did things they never considered with the newest materials ).

With that simple illustration of the difference between creativity and innovation, I'll ask you to think of them as a Venn Diagram that can overlap, but separately they still have value. A creative work and/or masterful work of art or technology can be extolled or appreciated regardless of whether it's new or innovative. An innovative work should be appreciated and differentiated because it brings something new to the world for further exploration and problem solving.

If we can learn to see the difference, then we can further appreciate the intersection of all three. We can also better understand how to foster each of them. Technology has advanced so greatly that we can easily "create" software solutions quickly today without mastery. We can do the same with paintings. For some problems, that ease of creating raw, imperfect work has it's benefit because it can allow for greater experimentation. So there's no benefit in forcing the three terms together.

I think the interwoven use of the two terms is the result of the new meaning of creativity. Let's be careful about that. Innovation is more than creativity - innovation is not only new, it's potentially disruptive and groundbreaking. It's noteworthy because of what it changes.

Song of the Angels by William-Adolphe Bouguereau

For more on this topic: I think the books and resources below nail it. Please let me know if you find any additional resources I should include in my own learning.

The Innovators by Walter Isaacson
Recently published in 2015, this book is a fun history lesson about the innovators that led to the creation of modern computers. The grouping of stories illustrates how a series of inventors and innovative thinkers influenced each other and virtually collaborated (or shared ideas) and competed with each other through the ages. There is much more to the book than I can capture in a paragraph but I couldn't stop reading and hope you enjoy it too.

Innovators Dilemma by Clayton M. Christensen
Fantastic and now classic book on the challenges that prevent organizations from innovating and engaging disruptive technology. Christensen published a more recent book The Innovator's Solution.

Innovator's Hypotheseis by Michael Schrage
In response to the challenge of innovation Schrage proposes that innovative teams work in parallel groups he calls 5x5. Leveraging what I consider the power of cooperation in an iterative cycle with a feedback loop and constraints this is the first published format that resonates with my own past experience of how creativity and innovation actually *work*.

The Art of Innovation Tom Kelley and Johnathan Littman
Also a classic for those of us who are into "design thinking". IDEO as an organization are experts in the process of creative and innovative thinking.

The Best Definition of Innovation by Scott Burkun
While updating this post, I found this refreshingly confronting article by Scott Burkun imploring us to use the word sparingly and he provides his definition of innovation in order to distinguish it.

Creating a Kubernetes Cluster With kube-aws

Kubernetes is one of the most promising container orchestration frameworks, because it is portable—not tied to a particular cloud provider (even though Google created it)—and because it is full-featured.

Unfortunately, creating a Kubernetes cluster is a little bit complex. The Kubernetes project currently provides two ways to create a cluster: (1) the kube-up script, which can be accessed from https://get.k8s.io, and (2) the kops tool, which has great potential but is relatively immature at this point. You can also create a cluster by hand, if you know what you are doing. But if you are working in AWS, then the best way currently, in my opinion, is by using the CoreOS “kube-aws” tool.

What Is a Kubernetes Cluster?

A cluster is just a collection of machines that are managed by a Kubernetes cluster controller. The machines can be real machines or virtual machines (VMs). Using real machines is more efficient since one then does not have the overhead of a hypervisor, which is redundant when using containers. (It remains to be seen what will become of the large commercial hypervisor-based VM ecosystem that exists today.)

If you are a software developer in a large organization, you will likely already have a cluster to deploy to—it would have been created for you. Alternatively, some teams prefer to create a cluster for each application, or for groups of applications. Regardless, someone has to create the cluster, and if you are working on your own, or in a self-sufficient team, then you do.

As I said above, I recommend the CoreOS kube-aws tool for creating Kubernetes clusters. Instructions for obtaining and using kube-aws can be found here. However, those instructions, while excellent overall, leave a few details out. Below I provide some tips that will hopefully fill in the gaps.

AWS Steps

I generally do all of my command line work on an AWS node, and I recommend that you do that too for setting up a Kubernetes cluster in AWS. Create a Linux VM in AWS, ssh into that instance, and then perform the steps described in the kube-aws instructions, with the tips below.

Obtain kube-aws

Tip: To see the full list of options for gpg2, type,

gpg2 --dump-options

Tip: To download kube-aws using curl, use the -L and -O options:

curl -L -O https://github.com/coreos/coreos-kubernetes/releases/download/v0.8.2/kube-aws-linux-amd64.tar.gz

Configure AWS Access

Tip: The AWS command line tools will have to be installed on your system. See here if you need to install them.

Tip: Instead of creating ~/.aws/credentials and ~/.aws/config files, you can set these environment variables:

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

Configure Key-Based Access to AWS Instances

Tip: Create an AWS EC2 SSH key pair, if you do not already have one. You give the key pair a name when you create it, and the public key is retained by AWS. You can then access it via the “Key Pairs” section of the EC2 console.

When a key pair is created, you will be instructed to download the public and private key pair and store the private key securely—either in your ~/.ssh directory, or in a project-specific location, and it should have 0400 access. You will specify the name of this key pair when you configure the Kubernetes cluster, and Kubernetes will use AWS to install the public key on each machine in the cluster. The presence of that key on each machine enables you to perform ssh -i <private-key-location> <userid>@<host> to connect to the machines that were created.

Tip: Create an SSL/TLS key pair. AWS provides a tool/system called KMS for this purpose. This is used for establishing SSL/TLS connections between users of your application and your Kubernetes load balancer.

Tip: In order to create the domain name that kube-aws requires as the name of the cluster controller, you will have to have an existing DNS domain so that you can create the cluster controller’s domain as a subdomain. To create that existing domain in AWS, you can use the Route 53 service: create a “Hosted Zone”, for example mydomain.com, and then for the Kubernetes cluster controller, specify a new subdomain, such as cluster.mydomain.com. kube-aws will automatically create the subdomain.

Create Cloud Formation Assets

kube-aws works by creating an AWS Cloud Formation file. Cloud Formation is AWS’s native computing environment orchestration tool. This is a good thing because by using Cloud Formation, kube-aws benefits from the maturity and robustness of Cloud Formation.

Create Directory For the Cluster Configuration

When creating a cluster, you keep the cluster’s configuration files in a directory on the machine from which you plan to administer the cluster. This is similar to other environment creation tools such as vagrant. Create a directory for your cluster:

mkdir mycluster

cd mycluster

Create Configuration File

Tip: If you want kube-aws to create the subdomain cluster.mydomain.com off of the existing domain mydomain.com, your AWS region is us-west-2, you would like the cluster to be created in zone us-west-2b of region us-west-2, your SSH key pair name is MyKeyPair (in which case, you will have downloaded a file called MyKeyPair.pem from AWS), and your SSL/TLS key (see “Encryption Keys” in the AWS “Identity and Access Management” console) ARN is arn:aws:kms:my-region:xxxxxxxxxxx:key/xxxxxxxxxxxxxxxx,

kube-aws init \

--cluster-name=mycluster \

--external-dns-name=cluster.mydomain.com \

--region=us-west-2 \

--availability-zone=us-west-2b \

--key-name=MyKeyPair \

--kms-key-arn="arn:aws:kms:my-region:xxxxxxxxxxx:key/xxxxxxxxxxxxxxxx"

Tip: The cluster-name must be unique within your AWS account.

Tip: The second level domain name of the subdomain specified by the external-dns-name—i.e., the “mydomain.com” part of cluster.mydomain.com—must be a valid registered DNS name. Merely creating a Hosted Zone in AWS for that domain will not be sufficient. You must create the hosted zone for the domain name (e.g., mydomain.com) in AWS, and also register the domain name with a domain registrar. AWS provides domain registration—see here.

Tip: Edit the resulting cluster.yaml file. E.g., set these to the values you prefer:

createRecordSet: true

hostedZoneId: <id>

workerCount: 2

workerInstanceType: m3.medium

workerRootVolumeSize: 30

tlsCADurationDays: 3650

tlsCertDurationDays: 365

where the <id> is the “Hosted Zone ID” of your AWS Route 53 hosted zone.

By default, kube-aws will create its own AWS Virtual Private Cloud (VPC). However, you can set the vpcId parameter in the cluster.yaml file if you want your cluster to use an existing VPC that you have created through other means.

Note: As of this writing, the generated cluster.yaml file uses the v1.3.6_coreos.0 version of the CoreOS image quay.io/coreos/hyperkube, which—according to CoreOS’s own scan, has 95 security vulnerabilities. You can select a different version by editing the kubernetesVersion value in the cluster.yaml file. However, even the most recent version (as of this writing) has 72 vulnerabilities. Some of the vulnerabilities involve denial of service attacks on the SSL implementation. This means that the cluster controller should only be deployed in a secure network and should not be directly accessible from the Internet—access from the Internet should only be via a proxy or a VPN/VPC.

Generate Cloud Formation

All we need to do is type,

kube-aws render

kube-aws validate

The output is a 600+ line stack-template.json file, which is a Cloud Formation file, and a kubeconfig file, which specifies which credentials to use when authenticating to your cluster.

Launch Cluster

kube-aws up

At this point, it displays a message, “Creating AWS resources. This should take around 5 minutes” - and indeed it does take that long, and gives you no feedback while it is working, but you can see progress if you go to the AWS Cloud Formation console, where you should see the “stack” listed with a “CREATE_IN_PROGRESS” message. When it finished the status should change to “CREATE_COMPLETE”.

When it completes you should also have command line output like this:

Creating AWS resources. This should take around 5 minutes.

Success! Your AWS resources have been created:

Cluster Name:    mycluster

Controller IP:    100.100.100.100

The containers that power your cluster are now being downloaded.

You should be able to access the Kubernetes API once the containers finish downloading.

If you then go to your AWS EC2 instance console, you should now see something like what is shown in the Figure below. Notice the first two instances: these are what I got when I launched a cluster for ScaledMarkets’ safeharbor service. If I had set the workerCount to be more than one, then there would be that many instances of kube-aws-worker.

Verify that the Cluster Is Accessible

You can now perform Kubernetes commands to manage your cluster. The primary command is the kubectl command. I launched my cluster on an x86-64 Centos 7 system, so I put the /kubernetes/platforms/linux/amd64 directory in my path. To verify access to the cluster, try

kube-aws status

That should give you something like this:

Cluster Name:    mycluster

Controller IP:    100.100.100.100

To print descriptions of the nodes in the cluster, use the kubectl command:

kubectl --kubeconfig=kubeconfig get nodes

Assuming that your DNS domain is registered and that there is an AWS Hosted Zone for it, you should get something like this:

NAME                                       STATUS                     AGE

ip-10-0-0-156.us-west-2.compute.internal    Ready                      2m

ip-10-0-0-50.us-west-2.compute.internal    Ready,SchedulingDisabled    2m

Deleting a Cluster

The command,

kube-aws destroy

deletes the cluster. However, it does not give you any feedback on progress. If you want to see that, go to your AWS Cloud Formation console, and you should see the Cloud Formation “stack” created by kube-aws with the message, DELETE_IN_PROGRESS.

Unfortunately, kube-aws destroy does not always work: it often fails to remove things like the VPC, load balancer, and other things that kube-aws up created. Thus, you might have to go into the Cloud Formation console event list to look at the messages and see which deletions failed, then then delete those things manually.

Using the Kubernetes Dashboard

Kubernetes has a nice dashboard for viewing and managing your cluster. However, your cluster is in AWS. If you are SSH-ing into a VM in AWS as I recommended, you will not be able to launch a browser there to view the dashboard. There are many ways to get around that. One is to boot a VM with X-Windows or Wayland and access that remotely. I have not tried that, so I can’t give you advice there. Another approach is to launch the AWS-based cluster from your local machine instead of from a VM in AWS as I have recommended. I don’t like to execute projects on my local machine because then I muck up my local machine—I like to always work from a VM that I can then blow away, using a separate VM for each project—but you can also avoid mucking up your local machine by creating a local VM with a console (“headed”) and working from that. It should work fine and be secure, but I don’t work that way. Finally, you can use run the Kubernetes proxy in AWS and access the proxy via the VM’s public IP address. To do that, type this on your AWS VM command line:

kubectl --kubeconfig kubeconfig proxy --accept-hosts="^*$" --address="<your-AWS-VM-private-IP-address>"

where <your-AWS-VM-private-IP-address> is the private IP address of your AWS VM. You can obtain that from the EC2 instance console. Then add port 8001 to the security group for your AWS VM, allowing any IP address for egress, but limiting to your local IP address for ingress—that is the “Source” setting in the AWS Security Group.

You can then view the dashboard at,

http://<your-AWS-VM-public-IP-address>:8001/ui

where <your-AWS-VM-public-IP-address> is the public IP address of your AWS VM.

The dashboard should look something like what is shown below.

You can click on the various dashboard elements to drill down—you can even edit a pod’s configuration and restart the pod, but I have not tried that. If you click on a pod, you will then see a detail page, showing the pod’s full name—the name shown on the dashboard main page is only an abbreviated name. Importantly, you can use the dashboard to find out the generated names of your pods, so that you can then perform actions such as attaching to a container:

kubectl --kubeconfig=kubeconfig attach <pod> -c <container>

Note that the kubectl proxy command blocks—canceling it with ^C will terminate the proxy—so you will probably want to run it with something like,

nohup kubectl --kubeconfig kubeconfig proxy --accept-hosts="^*$" --address="<your-AWS-VM-private-IP-address>" > log.out 2> log.err < /dev/null &

Note: Using the proxy is not a secure setup, because while you have restricted dashboard access to your IP address, IP addresses can be spoofed, and the proxy uses HTTP (not HTTPS). Therefore, you should only use the proxy from within a VPC/VPN or secure local network. However, when working in AWS I strongly recommend always working within a VPC/VPN anyway.

Conclusion

Once a cluster has been defined, we have one command to launch an entire cluster of machines, and we can administer that cluster with the kubectl command. The hard part was in creating the AWS keys and config files, but now that that has been done, we can create additional clusters with ease. Thus, we now have a repeatable and reliable process for standing up Kubernetes clusters—using AWS’s native features.

Now what? We have a cluster, but what can we do with it? That will be the topic of the next article, “Deploying a Kubernetes Service In Three Easy Steps”, in which I explain how to define and deploy an application configuration with Kubernetes.