The microservice design approach pertains to architecture, whereas DevOps pertains to how you build, test, and deliver—these should therefore be separate considerations, right?
No. Back in the 1980s I worked in the microchip design and testing field, and it was essential—even back then—to design chips in a way that made them testable: "design-for-testability" was—and is to this day—a major consideration for design. In fact, a good percentage of a chip's circuitry was there only to enable the chip to be tested.
This is not new, and designing software for testability is even more important today than it was only a decade ago, because today, if you want to "shift left" your integration testing, you will need to be able to create small-footprint transient integration test environments. That means that you are deploying your apps again and again, and if your apps are big and monolithic, then not only will deployment take a long time, but the apps will also have a large footprint, which translates into money.
Consider, for example, that one of your application components is a large SOA based system that provides an "information layer", and that this SOA layer can only be deployed as a single unit—i.e., all-or-nothing. If one of your teams is building an app that uses that SOA layer, then in order for the team to create an integration test environment on demand, they will have to dynamically deploy the entire SOA based system into their environment.
Why do they need to do this to test? They need to because they want to test changes to their app, and not disturb anyone else. That's why they need a test environment of their own. It only needs to be their own for the duration of their test. They can't use a shared test instance of the SOA layer, because the SOA layer contains databases, and therefore is stateful, and so the tests change the SOA layer. (Even worse, the features being tested might even included code changes to the SOA layer.) Any change to the SOA layer would affect other teams that are using that SOA layer for their testing, and that is why a test environment must be exclusive to the test agent—it cannot be shared (at least for the duration of the test run). In other words, a test environment needs to be isolated.
Many DevOps teams achieve isolation by spinning up an environment on demand and deploying all of the various application components to that environment, running the tests, and then destroying the environment. Thus, if one or more of the components is a large, monolithic system, it is difficult and expensive to use a dynamic environment testing strategy. This is a very significant handicap.
Microservices are small, independently deployable components, and so when one needs to perform an integration test that involves microservices, one can select only those needed for the test, deploy them, run the tests, and then destroy. Of course, one must consider dependencies among the microservices: that is part of the test planning that goes into the development of a feature or story. Essentially, one must design what the "test bench" should be for the system-under-test—i.e., what other components are needed besides the component that has been modified and is being tested, to enable the test to be performed.
The small footprint and independent deployability of microservices is therefore a major enabler for shift-left integration testing. If one has monolithic components, then shifting integration testing left, to the team (or to the individual developer) is very difficult, and one usually has to fall back to a shared integration test process whereby all components are deployed at regular intervals into that environment and tests are run. In such a process, there are usually a-lot of failed tests—the test run does not stay "green", and so determining the cause of failures is difficult. Also, in that approach, it is necessary to use "feature toggles" to turn off features that are not yet complete, since incompletely integrated features will cause other tests to fail that used to pass, putting those features in doubt. Using feature toggles is complicated, and it is a messy situation. That is why DevOps teams try to "shift left" and perform early integration testing before they make a feature visible to the downstream test pipeline; but to do that, your components need to be small enough to be able to deploy them frequently into local integration test environments.
Friday, July 28, 2017
Saturday, June 24, 2017
Why DEV, INT, and QA environments get in the way of DevOps
Organizations that develop software often have static development and test environments, with names like "DEV", "INT", "QA", and so on. These are usually statically provisioned environments. They fit the paradigm of "servers as pets and cattle".
DevOps does not work that way, and if you continue to use names like that, you will find it very hard to transition to a DevOps approach. The modern paradigm is to think of servers as something that get created on demand. In fact, if you use "platform-as-a-service" or "serverless" services from your cloud provider, you don't even worry about servers (or containers)—the cloud pushes that behind the scenes for you, and all you worry about is your application.
The problem is, many organizations have deeply embedded their internal "DEV", "INT", "QA" terminology into their governance processes and their software development and testing methodologies, and getting them out is extremely difficult. Thus, if one wants to, say, stand up an environment on demand for testing, whenever needed, people in the organization ask, "Are you doing that in INT, or in QA?" Such a question makes no sense, because in a modern development pipeline there is no INT or QA environment: environments are created on demand. You might characterize a test suite as an "INT" or "QA" test suite, and you might even characterize the environment template for that test as a "INT" or "QA" environment template, but the environment itself does not exist until you deploy to it for testing. Also, you should not leave test environments sitting around after you have used them: doing that is expensive and wasteful.
All this sounds pretty harmless, but it is not—far from it. Static test environments really cannot support the modern version of Agile processes, which rely on a feature-based approach to development. Consider the gold standard of a traditional Agile development process: continuous integration, aka "CI". CI requires that the CI job—perhaps defined in a Jenkins server—has an environment that it alone has control of; otherwise, the tests that the CI job runs will not be reproducible: if someone else is doing things in that environment (e.g., doing manual testing, which leaves behind database data), then a CI test might pass one time, but run again it might fail—even if the application code didn't change. The CI process becomes untrustworthy.
Historically, development teams have run only their unit tests in their CI environment. That's fine for a single-team project; but things get sketchy really fast if you have multiple teams that are building a large system that has many different components. For integration testing of how the many components work together, Agile projects have historically used a single project-wide statically provisioned integration test environment. Because there was only one such environment, access to it had to be scheduled, or restricted to the "integration test team". If you think about that, it is really a waterfall process, wrapped around all of the project teams' individual Agile processes. Integration testing done this way is single-threaded. So when a programmer makes a change, they can't tell if they have broken something in the system as a whole until the integration test team runs the integration tests. If those tests are manual, then it might be days (or weeks, if integration test deployment is manual) before the programmer gets feedback. That's not very Agile.
This was as good as we could do until cloud computing came along. Clouds made it possible to stand up entire test environments on demand. That made it possible to "shift left" the integration testing—to move it into the CI cycle (or even better, to the individual programmer), so that the team's Jenkins job runs integration tests. To be clear, in this approach each team's Jenkins job deploys the entire system (isolated from any other team's) and then runs system-wide integration tests against that. (See this article for more information on how continuous delivery methods revise traditional Agile practices.)
This is where the DEV, INT, and QA thinking gets in the way. If the organization has a QA group, they will want a "QA environment" where they can run their integration tests. But you say, "We create that on demand", and then they get confused, because tradition has linked three things that are actually independent: (1) a class of tests (integration tests), (2) a place where they get executed; and (3) who creates or performs those tests.
It gets worse. Because people equate these things in their thinking, they can't get their head around the idea that DevOps teams don't "push code to an environment", and for that reason, they also can't understand how DevOps teams can perform integration tests on features that have not yet been merged into the main codebase. This is because static environment thinking says that everyone has to "put their code somewhere", and that implies that all the features for the release are present in the code—otherwise, many tests will not pass. But DevOps teams are usually set up as "feature teams", meaning that they work on a cross-component feature at a time, modifying any of the system's components, and they integration test that cross-component feature. That integration testing happens before they merge the feature's code changes into the shared development code branch for each component. Thus, they don't "put their code into an environment". Rather, they integration test a set of features changes—spanning multiple components—and then merge the changes. Then, if another team wants to integration test (either another development team, or a QA team), they pull the latest merged code, and they will obtain only completed (working) features.
Sometimes I hear cloud vendors speak in terms of "creating your INT and QA environments", and when I ask them about it, they say that they are trying to "bridge to something the customer understands". However, that is holding those customers back. The DEV, INT, QA terminology and thinking is a major impediment to letting go of the static paradigm, and letting that go is foundational for understanding DevOps. Start to think of "INT" and "QA" as kinds of environment—not as specific environments that are sitting there, waiting for you. Even better, start to think in terms of kinds of tests that you need to run, and the environment configurations that you need for each of those, and then create a template or script to provision each of those types of environment on demand. See this article series for more on defining a testing strategy.
DevOps does not work that way, and if you continue to use names like that, you will find it very hard to transition to a DevOps approach. The modern paradigm is to think of servers as something that get created on demand. In fact, if you use "platform-as-a-service" or "serverless" services from your cloud provider, you don't even worry about servers (or containers)—the cloud pushes that behind the scenes for you, and all you worry about is your application.
The problem is, many organizations have deeply embedded their internal "DEV", "INT", "QA" terminology into their governance processes and their software development and testing methodologies, and getting them out is extremely difficult. Thus, if one wants to, say, stand up an environment on demand for testing, whenever needed, people in the organization ask, "Are you doing that in INT, or in QA?" Such a question makes no sense, because in a modern development pipeline there is no INT or QA environment: environments are created on demand. You might characterize a test suite as an "INT" or "QA" test suite, and you might even characterize the environment template for that test as a "INT" or "QA" environment template, but the environment itself does not exist until you deploy to it for testing. Also, you should not leave test environments sitting around after you have used them: doing that is expensive and wasteful.
All this sounds pretty harmless, but it is not—far from it. Static test environments really cannot support the modern version of Agile processes, which rely on a feature-based approach to development. Consider the gold standard of a traditional Agile development process: continuous integration, aka "CI". CI requires that the CI job—perhaps defined in a Jenkins server—has an environment that it alone has control of; otherwise, the tests that the CI job runs will not be reproducible: if someone else is doing things in that environment (e.g., doing manual testing, which leaves behind database data), then a CI test might pass one time, but run again it might fail—even if the application code didn't change. The CI process becomes untrustworthy.
Historically, development teams have run only their unit tests in their CI environment. That's fine for a single-team project; but things get sketchy really fast if you have multiple teams that are building a large system that has many different components. For integration testing of how the many components work together, Agile projects have historically used a single project-wide statically provisioned integration test environment. Because there was only one such environment, access to it had to be scheduled, or restricted to the "integration test team". If you think about that, it is really a waterfall process, wrapped around all of the project teams' individual Agile processes. Integration testing done this way is single-threaded. So when a programmer makes a change, they can't tell if they have broken something in the system as a whole until the integration test team runs the integration tests. If those tests are manual, then it might be days (or weeks, if integration test deployment is manual) before the programmer gets feedback. That's not very Agile.
This was as good as we could do until cloud computing came along. Clouds made it possible to stand up entire test environments on demand. That made it possible to "shift left" the integration testing—to move it into the CI cycle (or even better, to the individual programmer), so that the team's Jenkins job runs integration tests. To be clear, in this approach each team's Jenkins job deploys the entire system (isolated from any other team's) and then runs system-wide integration tests against that. (See this article for more information on how continuous delivery methods revise traditional Agile practices.)
This is where the DEV, INT, and QA thinking gets in the way. If the organization has a QA group, they will want a "QA environment" where they can run their integration tests. But you say, "We create that on demand", and then they get confused, because tradition has linked three things that are actually independent: (1) a class of tests (integration tests), (2) a place where they get executed; and (3) who creates or performs those tests.
It gets worse. Because people equate these things in their thinking, they can't get their head around the idea that DevOps teams don't "push code to an environment", and for that reason, they also can't understand how DevOps teams can perform integration tests on features that have not yet been merged into the main codebase. This is because static environment thinking says that everyone has to "put their code somewhere", and that implies that all the features for the release are present in the code—otherwise, many tests will not pass. But DevOps teams are usually set up as "feature teams", meaning that they work on a cross-component feature at a time, modifying any of the system's components, and they integration test that cross-component feature. That integration testing happens before they merge the feature's code changes into the shared development code branch for each component. Thus, they don't "put their code into an environment". Rather, they integration test a set of features changes—spanning multiple components—and then merge the changes. Then, if another team wants to integration test (either another development team, or a QA team), they pull the latest merged code, and they will obtain only completed (working) features.
Sometimes I hear cloud vendors speak in terms of "creating your INT and QA environments", and when I ask them about it, they say that they are trying to "bridge to something the customer understands". However, that is holding those customers back. The DEV, INT, QA terminology and thinking is a major impediment to letting go of the static paradigm, and letting that go is foundational for understanding DevOps. Start to think of "INT" and "QA" as kinds of environment—not as specific environments that are sitting there, waiting for you. Even better, start to think in terms of kinds of tests that you need to run, and the environment configurations that you need for each of those, and then create a template or script to provision each of those types of environment on demand. See this article series for more on defining a testing strategy.
Saturday, January 21, 2017
Inserting DevOps Into a Not-Very-Agile Organization
This is an account of a recent experience at a very large company that has a mix of traditional (waterfall) and some partly Agile IT projects, and a tiny bit of DevOps here and there. Tricia Ratliff and I were asked to take a new project and
help the project's manager to “do Agile right”—meaning use DevOps and anything else that makes sense.
It worked. In two months, a team that had almost no Agile experience and no knowledge of any automation tools was able to,
Another benefit that we have seen is that work is going faster and faster, as the team becomes more proficient in using the tools and refines its process. It is almost impossible to compare the productivity of different Agile teams, but I personally estimate that this team is at least two times as productive as the other non-DevOps “Agile” team that I am working with at the same customer location—even though this team's application is more difficult to test because it has a UI, whereas the other team's application does not. (I actually think that the DevOps team might be three or four times as productive as other teams at this client.)
What makes the team so productive is that the team is able to work in a “red-green cycle” (see here, and here), whereby they code, test, code, test, code, and test until the tests pass, and then move on to the next Agile story. The red-green cycle is known to the Test-Driven Development (TDD) community, but we use it for integration testing, locally (on our laptops). It is made possible by the use of the ATDD process, in which automated tests are written before the associated application code. (See here for a comparison of TDD and ATDD.) I will note that the team's application is a Web application, with automated tests written for each bit of UI functionality before the associated application code is written.
This is important: the ability of developers to use a red-green cycle is a game changer, in terms of productivity, code quality, and code maintainability; and we are doing this for integration testing—shifting it “left” into a red-green cycle on the developer's laptop.
It is important to emphasize that there is no single one way to do DevOps. By its nature, all of this is contextual. It is not possible to define a “standard process” for how teams should provision, develop/test and deploy. However, it is possible to devise common patterns that can be re-used, as long as a team feels free to adjust the pattern. If you want to have a learning organization—and having a learning organization is essential for effective Agile and DevOps—then teams need to have the ability to experiment and to tailor their process.
We did not use “baby steps” in our adoption of DevOps for this project. Rather, we undertook to change everything from the outset, since ours was a green field project, and we did not want to entrench legacy practices. We used an Agile coach (Tricia) and DevOps coach (me) to explain the new techniques to the team and make a compelling case for each engineering practice, but nothing was mandated. The coaches facilitated discussions on the various topics, and the team developed its own processes as a group.
By far the most impactful thing we did was to establish a complete end-to-end testing strategy from the outset. We included the entire team and all extended team members in the development of this strategy. The team consisted of all of the developers and testers (including technical lead and test lead), as well as the team's manager. The extended team members who we included were the team's application architect, an expert from the organization's test automation group, and testing managers representing integration testing and user acceptance testing. (We would have liked to include someone from the infrastructure side, but at that time it was not clear who that should be.) Devising the testing strategy began with a four-hour session at a whiteboard. The output of that session was a table of testing categories, with columns as follows: (1) category of test, (2) who writes the tests, (3) where/when the tests will be run, and (4) how we plan to measure sufficiency or “coverage” for the tests. (The format of this planning artifact was based on prior work of mine and others, which is described in this four-part article.)
We did not try to fill in all of the cells in our testing strategy table in our first session, but having the rows defined provided the foundation for our DevOps “pipeline”. Some of the testing categories that we included in our table were (a) unit tests, (b) behavioral integration tests, (c) end-to-end use case level tests, (d) failure recovery tests, (e) performance and stress tests, (f) security tests and scans, (g) browser compatibility tests, and (h) exploratory tests. Our “shift left” testing philosophy dictated that we run every kind of test as early as possible—on our laptops if it makes sense—rather than waiting for it to be run only downstream by a Jenkins job. In practice, we run the unit tests, behavioral integration tests, use case level tests, and basic security scans on our laptops before committing code to the main branch, which triggers a Jenkins “continuous integration” (CI) job that re-runs the unit tests, builds a container image, deploys a system instance (service/pod) to our OpenShift cluster, and then re-runs the behavioral and end-to-end tests against that system instance. Our Jenkins CI job therefore verifies that the system, as deployed, will work the same as the system that we tested on our laptops, because we use the same OpenShift template for deploying a test instance as we use for deploying a production instance. We plan to add Jenkins jobs for non-functional testing such as performance testing, deep security scanning and automated penetration testing: these will each begin by deploying a system instance to test, and will end by destroying that system instance: thus, no system instance is re-used across multiple tests, and we can perform any of these tests in parallel.
The issue of running tests locally is an important one. Too often DevOps is depicted or described as a sequence of tests run by a server such as Jenkins. However, that model re-creates batch processing, whereby developers submit jobs and wait. Shifting left is about avoiding the waiting: it is about creating a red-green cycle for each developer. Thus, real DevOps is actually about shortening that sequence of Jenkins jobs—perhaps even eliminating it. In an ideal DevOps process, there would be no Jenkins jobs: those jobs are a necessary evil, and they exist only because some tests are impractical to perform on one's laptop.
We spent two months preparing for our first Agile development iteration. During the two month startup period we requested tools, defined our work process, met with business and technical stakeholders, and received training. Most meetings involved the entire team, although there were also many one-on-one behind-the-scenes meetings with various stakeholders to talk through concerns. We arranged for an all-day all-team hands-on training on OpenShift from Red Hat, and we also arranged for a four-hour hands-on Cucumber and Selenium training session from an internal test automation group. Both of those training sessions were essential, and we saw an immense jump in overall understanding among the team after each of those training sessions.
However, it was not all smooth.
One mistake we made is that we did not make it clear to the Testing resources unit that the test programmers would need to arrive at the same time as our developers. The late arrival of the test programmers was a significant problem during the initial iterations, because when they arrived they were not up to speed on how we were doing things or on the application stories that the team and Product Owner had collaboratively developed. This problem was exacerbated by the fact that the new arrivals could not participate in the actual work until they had access to the team’s source code repo, but per organization policy they could not be granted access to that until they had had corporate git training, and there was no git training scheduled until the next month. Fortunately the project manager was eventually able to escalate this issue and get the test programmers git access, but during iteration one the application developers wrote most of the automated tests because the test programmers did not have git access.
Bureaucratic obstacles like this were the norm, and so it was extremely important that the team coaches and project manager stay on top of what was in the way each day, and escalate issues to the appropriate manager, explaining that in an Agile project such as ours, a two week delay was an entire iteration, and so obstacles had to be removed in hours or days—not weeks. Some of the functional areas that we had extended conversations with were application architecture, systems engineering, data architecture, the infrastructure engineering team, and others, because not only did each of these functions have authority over different aspects of our project pipeline, but we needed their help to enable us to connect our application into the enterprise infrastructure.
For example, I recall explaining to the assigned infrastructure engineer that we did not need “an environment”, nor did we need for him to install an application server for us, since we were going to receive access to a cluster that was being created, into which we would dynamically create containers built from base images that contain all of the components that we need (such as an application server), and it took three conversations before he grasped that. I still have trouble explaining to people in the organization that this team does not have any “environments” because we use dynamic environment provisioning via an OpenShift cluster. This has presented communication challenges and policy confusion because much of the organization's procedures and governance rules are phrased in terms of control of environment types, such as the “integration test environment” and the “user acceptance environment”. For us, however, an environment does not exist until we perform a deployment, and then after we run the tests, we destroy the environment to free up the resources.
Some of the tools that we needed were not available right away, because various enterprise groups were busy setting them up or testing them for security, so team members had to download them at home to try them out. These external groups included (1) the new OpenShift cluster team, (2) the Infrastructure Engineering team which was creating the JBoss base image that we were to use; (3) the team that manages the internal software library repository; and (4) the team that was setting up the container image registry (Artifactory) that we were to use. Once they understood what we were trying to do, all of these groups did their best to get us what we needed, but it took time since these things were being done for the first time.
Another significant challenge was that the organization requires its software developers to work on its “production” network, using “production” laptops—a practice that makes it very difficult for software engineers who need to be able to download new tools on a frequent basis to try them out: one is not allowed to download software from the Internet into the production network. (I have advocated for having a less secure sandbox network that all developers can access via a remote desktop.) It also turned out that the laptops, which use Windows, have a browser security tool called Bromium which uses virtualization, and it was preventing our team from being able to run virtual machines on their laptops—something that is required to be able to run Linux containers under Windows. We had to have Bromium technical support spend several weeks at our site, coming up with a configuration that would allow our team to run virtual machines. Diagnosing that problem and arranging for the solution delayed the team's learning about Linux containers and OpenShift by almost two months. (Also, using Windows laptops for building software that will be deployed in Linux containers makes absolutely no sense, and has the side effect that developers do not become familiar with the target environment—Linux).
It worked. In two months, a team that had almost no Agile experience and no knowledge of any automation tools was able to,
- Learn Acceptance Test-Driven Development (ATDD) and the associated tools (Cucumber, Selenium).
- Learn to use virtualization and Linux (Docker) containers, both on their laptops and in our data center.
- Learn to use OpenShift—Red Hat's enhanced version of the Kubernetes container orchestration framework.
- Learn a NoSQL database (Cassandra—project has since switched to MongoDB), having no prior experience with NoSQL databases.
- Get productive and start delivering features in two-week Agile iterations, using a fully “left-shifted” testing process in which developers get all integration tests to pass before they merge their code into the main development branch.
The Benefits We Have Seen
One of the benefits we have seen is that the team is not dependent on anyone or anything outside our team except for our git server (TeamForge). For example, our OpenShift cluster and Jenkins server were down for two weeks awhile back, but the team was able to continue working and delivering completed stories, because they were able to perform integration tests locally on their laptops.Another benefit that we have seen is that work is going faster and faster, as the team becomes more proficient in using the tools and refines its process. It is almost impossible to compare the productivity of different Agile teams, but I personally estimate that this team is at least two times as productive as the other non-DevOps “Agile” team that I am working with at the same customer location—even though this team's application is more difficult to test because it has a UI, whereas the other team's application does not. (I actually think that the DevOps team might be three or four times as productive as other teams at this client.)
What makes the team so productive is that the team is able to work in a “red-green cycle” (see here, and here), whereby they code, test, code, test, code, and test until the tests pass, and then move on to the next Agile story. The red-green cycle is known to the Test-Driven Development (TDD) community, but we use it for integration testing, locally (on our laptops). It is made possible by the use of the ATDD process, in which automated tests are written before the associated application code. (See here for a comparison of TDD and ATDD.) I will note that the team's application is a Web application, with automated tests written for each bit of UI functionality before the associated application code is written.
 |
| Figure 1: The red-green cycle of test-first development. |
How We Did It
The shift to proper Agile and DevOps is not a small adjustment or substitution. It is a very large set of interdependent changes, involving new methods, new mindsets, and new tools. More on this later.It is important to emphasize that there is no single one way to do DevOps. By its nature, all of this is contextual. It is not possible to define a “standard process” for how teams should provision, develop/test and deploy. However, it is possible to devise common patterns that can be re-used, as long as a team feels free to adjust the pattern. If you want to have a learning organization—and having a learning organization is essential for effective Agile and DevOps—then teams need to have the ability to experiment and to tailor their process.
We did not use “baby steps” in our adoption of DevOps for this project. Rather, we undertook to change everything from the outset, since ours was a green field project, and we did not want to entrench legacy practices. We used an Agile coach (Tricia) and DevOps coach (me) to explain the new techniques to the team and make a compelling case for each engineering practice, but nothing was mandated. The coaches facilitated discussions on the various topics, and the team developed its own processes as a group.
By far the most impactful thing we did was to establish a complete end-to-end testing strategy from the outset. We included the entire team and all extended team members in the development of this strategy. The team consisted of all of the developers and testers (including technical lead and test lead), as well as the team's manager. The extended team members who we included were the team's application architect, an expert from the organization's test automation group, and testing managers representing integration testing and user acceptance testing. (We would have liked to include someone from the infrastructure side, but at that time it was not clear who that should be.) Devising the testing strategy began with a four-hour session at a whiteboard. The output of that session was a table of testing categories, with columns as follows: (1) category of test, (2) who writes the tests, (3) where/when the tests will be run, and (4) how we plan to measure sufficiency or “coverage” for the tests. (The format of this planning artifact was based on prior work of mine and others, which is described in this four-part article.)
We did not try to fill in all of the cells in our testing strategy table in our first session, but having the rows defined provided the foundation for our DevOps “pipeline”. Some of the testing categories that we included in our table were (a) unit tests, (b) behavioral integration tests, (c) end-to-end use case level tests, (d) failure recovery tests, (e) performance and stress tests, (f) security tests and scans, (g) browser compatibility tests, and (h) exploratory tests. Our “shift left” testing philosophy dictated that we run every kind of test as early as possible—on our laptops if it makes sense—rather than waiting for it to be run only downstream by a Jenkins job. In practice, we run the unit tests, behavioral integration tests, use case level tests, and basic security scans on our laptops before committing code to the main branch, which triggers a Jenkins “continuous integration” (CI) job that re-runs the unit tests, builds a container image, deploys a system instance (service/pod) to our OpenShift cluster, and then re-runs the behavioral and end-to-end tests against that system instance. Our Jenkins CI job therefore verifies that the system, as deployed, will work the same as the system that we tested on our laptops, because we use the same OpenShift template for deploying a test instance as we use for deploying a production instance. We plan to add Jenkins jobs for non-functional testing such as performance testing, deep security scanning and automated penetration testing: these will each begin by deploying a system instance to test, and will end by destroying that system instance: thus, no system instance is re-used across multiple tests, and we can perform any of these tests in parallel.
The issue of running tests locally is an important one. Too often DevOps is depicted or described as a sequence of tests run by a server such as Jenkins. However, that model re-creates batch processing, whereby developers submit jobs and wait. Shifting left is about avoiding the waiting: it is about creating a red-green cycle for each developer. Thus, real DevOps is actually about shortening that sequence of Jenkins jobs—perhaps even eliminating it. In an ideal DevOps process, there would be no Jenkins jobs: those jobs are a necessary evil, and they exist only because some tests are impractical to perform on one's laptop.
We spent two months preparing for our first Agile development iteration. During the two month startup period we requested tools, defined our work process, met with business and technical stakeholders, and received training. Most meetings involved the entire team, although there were also many one-on-one behind-the-scenes meetings with various stakeholders to talk through concerns. We arranged for an all-day all-team hands-on training on OpenShift from Red Hat, and we also arranged for a four-hour hands-on Cucumber and Selenium training session from an internal test automation group. Both of those training sessions were essential, and we saw an immense jump in overall understanding among the team after each of those training sessions.
However, it was not all smooth.
Obstacles We Overcame
We encountered many institutional obstacles. Our project manager was committed to using a DevOps approach, and was key because he was always ready to escalate an issue and get blockers removed. One of the myths in the Agile community is that project managers are not needed in Agile projects; yet in my experience a project manager is extremely important if the setting is a large IT organization with a-lot of centralization, because a project manager has indisputable authority within the organization and a person with authority is needed to advocate effectively for the team.One mistake we made is that we did not make it clear to the Testing resources unit that the test programmers would need to arrive at the same time as our developers. The late arrival of the test programmers was a significant problem during the initial iterations, because when they arrived they were not up to speed on how we were doing things or on the application stories that the team and Product Owner had collaboratively developed. This problem was exacerbated by the fact that the new arrivals could not participate in the actual work until they had access to the team’s source code repo, but per organization policy they could not be granted access to that until they had had corporate git training, and there was no git training scheduled until the next month. Fortunately the project manager was eventually able to escalate this issue and get the test programmers git access, but during iteration one the application developers wrote most of the automated tests because the test programmers did not have git access.
Bureaucratic obstacles like this were the norm, and so it was extremely important that the team coaches and project manager stay on top of what was in the way each day, and escalate issues to the appropriate manager, explaining that in an Agile project such as ours, a two week delay was an entire iteration, and so obstacles had to be removed in hours or days—not weeks. Some of the functional areas that we had extended conversations with were application architecture, systems engineering, data architecture, the infrastructure engineering team, and others, because not only did each of these functions have authority over different aspects of our project pipeline, but we needed their help to enable us to connect our application into the enterprise infrastructure.
For example, I recall explaining to the assigned infrastructure engineer that we did not need “an environment”, nor did we need for him to install an application server for us, since we were going to receive access to a cluster that was being created, into which we would dynamically create containers built from base images that contain all of the components that we need (such as an application server), and it took three conversations before he grasped that. I still have trouble explaining to people in the organization that this team does not have any “environments” because we use dynamic environment provisioning via an OpenShift cluster. This has presented communication challenges and policy confusion because much of the organization's procedures and governance rules are phrased in terms of control of environment types, such as the “integration test environment” and the “user acceptance environment”. For us, however, an environment does not exist until we perform a deployment, and then after we run the tests, we destroy the environment to free up the resources.
Some of the tools that we needed were not available right away, because various enterprise groups were busy setting them up or testing them for security, so team members had to download them at home to try them out. These external groups included (1) the new OpenShift cluster team, (2) the Infrastructure Engineering team which was creating the JBoss base image that we were to use; (3) the team that manages the internal software library repository; and (4) the team that was setting up the container image registry (Artifactory) that we were to use. Once they understood what we were trying to do, all of these groups did their best to get us what we needed, but it took time since these things were being done for the first time.
Another significant challenge was that the organization requires its software developers to work on its “production” network, using “production” laptops—a practice that makes it very difficult for software engineers who need to be able to download new tools on a frequent basis to try them out: one is not allowed to download software from the Internet into the production network. (I have advocated for having a less secure sandbox network that all developers can access via a remote desktop.) It also turned out that the laptops, which use Windows, have a browser security tool called Bromium which uses virtualization, and it was preventing our team from being able to run virtual machines on their laptops—something that is required to be able to run Linux containers under Windows. We had to have Bromium technical support spend several weeks at our site, coming up with a configuration that would allow our team to run virtual machines. Diagnosing that problem and arranging for the solution delayed the team's learning about Linux containers and OpenShift by almost two months. (Also, using Windows laptops for building software that will be deployed in Linux containers makes absolutely no sense, and has the side effect that developers do not become familiar with the target environment—Linux).
DevOps Is Not a Small Change
I mentioned earlier that the shift to proper Agile and DevOps is not a small adjustment or substitution. This is interesting because we asked for some changes to the IT process controls in order to enable us to do things the way we wanted to do them. From the point of view of the controls group, our process was almost the same as the standard process, but that is only because the IT control view of a software development process is so disconnected from the reality: our process is vastly different from how other Agile teams at this client work. Some of the differences are,- We created an integrated automated testing strategy without handoffs: other Agile projects at this customer have a sequential testing process, with different teams responsible for each.
- We use what is called a “build once, deploy many” approach, whereby we build a deployable image and store that, and that is deployed automatically to successive test environments for automated testing—many times per day.
- Developer laptops are their “DEV” environment—we do not have a shared “DEV” environment as other Agile teams at this client do. Thus, each developer tests with their own full stack, including their own test database instance, eliminating interference between testing by developers.
- We replaced manual integration testing with an industry standard “continuous integration” (CI) automated testing approach. This is unusual for this client.
- Analysts and testers write automated test specs using the well known Cucumber tool—they don’t run tests. The analysts focus mainly on the logical functions of the application, and the testers focus on nuances such as error cases and data requirements, and encode their understanding in the Cucumber test specs (feature files). This is widely known in the industry as Acceptance Test-Driven Development (ATDD).
- Test programmers and developers write automated tests that implement the test specs.
- Team members are allowed to switch roles, and one of our analysts has written test code. The only restriction that we have is that if you have written the test code for a story, someone else must write the application code for that story. This ensures that for a story to pass its tests, two different people must have the same understanding of the story's requirements.
- Our test lead supervises this process to make sure that this rule is followed, and that tests are written in a timely manner. Our Scrum Master makes sure that test coding and app coding tasks are maintained on our physical planning wall, as well as in our Agile tool (VersionOne).
- As a result of the above, a developer never waits for a tester: tests are written and developers run the tests—not the testers. Thus, a developer codes a story, runs tests, fixes the defects, re-runs the tests, and so on until there are no defects. In other words, we use a red/green cycle.
- The organization's “User Acceptance Testing” (UAT) team is a testing partner—not a handoff. They provided two members of our team, and those testers write our end-to-end use case level tests, which get run alongside all of our other tests.
- UAT adds its tests to the project repo, and so the developers can run the UAT tests locally on their laptops and in their CI environment—there is no delay.
- The development team writes its own CI build scripts. Other projects at this client typically receive their Jenkins build scripts from an integration team.
- The development team writes the production deployment OpenShift templates and scripts, tests those, and uses those to deploy to each test environment.
- All functional tests are run in each environment type (laptop, CI).
- Developers run security scans and code quality scans locally before they check in their code, and review the scan results produced by the CI build as an integrated task for the development of each story.
- All of our CI test environments are created from scratch for each test run, using the OpenShift template that the team coded. Thus, CI tests are always “clean”.
- The CI test database is cleared and data is loaded from scratch prior to each behavioral test case. Thus, there is never an issue with data left over from a prior test, or with a tester waiting for someone to be done testing a database table. (Eventually we plan to integrate the database into our OpenShift pod configuration, so that a fresh database will be deployed each time we deploy the application for a test run, but we are waiting for an approved container image for our database.)
- We don't have a DBA on our team, because we are using a NoSQL database, which has no schema. We work with an analyst who maintains a logical data model, but the model is maintained during development—not ahead of time. Our CI tests are all regression tests, so any breaks caused by changes to the data model are caught immediately.
- We have a very minimal need for “defect management”, because most tests are automated, and so most test results are visible in a dashboard in our Jenkins server. A developer does not check in a story’s code until it passes all of the behavioral tests locally for that story, and all affected tests are still passing. However, our exploratory testing is manual (by definition), and we record issues found during those test sessions. Exploratory testing is for the purpose of discovering things that the test designers did not anticipate, as well as for assessing the overall usability of the application.
- We don’t accept any story as done unless it has zero known defects. Thus, our build images that are marked as deployable typically have zero known defects.
- Everyone on the team (analyst, tester programmer, developer, coaches, Scrum Master, PM) has write access to all of the team’s tools and repositories (git repos, images, Jenkins project, OpenShift project, VersionOne project, Sharepoint project), and everyone received training in all of those tools. However, we feel we have devised a secure process that leverages the built-in secure change history that these tools provide.
- Our Product Owner reviews test scenarios as written in the “Cucumber” test specs, to ensure that they meet the intent of the associated Agile story's acceptance criteria. To do this, the Product Owner accesses test specs from the git source code control system. There are no spreadsheets—all artifacts are “executable”. To learn how to do this, the Product Owner—who is a manager who works in a business area—attended git training.
- Our iterations are two weeks—automated testing makes a three week iteration unnecessary: three weeks is the norm for other Agile teams at this client. Yet, our team seem to produce significantly more completed work per iteration than other teams.
- We only need minimal data item level tests, since our acceptance test driven process, for which we measure coverage, actually covers data items.
- We revise governance artifacts during each iteration, so that they are always up to date.
Sunday, October 16, 2016
Understanding Innovation, Creativity and Mastery
To illustrate the difference I will use realist oil painting, another genre I'm familiar with.
As a practiced artist, I have the tools and skills to paint oil portraits that are completely customized. What makes them creative is that they are designed and crafted as a one of a kind object. By capturing a person's likeness in paint by hand, a conservator can not only distinguish between my work and that of other artists but they can also distinguish between two paintings by me of the same subject - because they are new and "one-of-a-kind".
Merriam Webster's definition of Creativity; "The ability to make new things or think of new ideas"
Those painted images are creative because they were made as new things. That particular person has never been painted before and their image has not been previously captured in that way. But I can say that creativity doesn't necessarily make all of my custom images "innovative." It is my great hope that they are beautiful and valuable so they will become a legacy for future generations. But even if a painter has "mastered" a technique from the past, that established, time tested technique is not innovative - but it can be used to paint a creative work of art. Let's also appreciate the value of "mastery" both with or without creativity. Those same techniques can be used by a skilled painter to create a work of art that is not 'creative' by the dictionary definition but it is still beautiful and masterful. That masterful work is valuable because of the time and skill required to make it.
Now let's consider how creativity and mastery can be combined in the work of innovation. Imagine that same work of art but this time the painter who has mastered a given technique goes beyond earlier techniques and does something groundbreaking and completely new. That painter may use an old technique and combine it with newly discovered pigment or substance that creates an effect nobody has ever seen before. The effect might be dramatic or so subtle that only the innovator knows why it works. ( Yes, this is still possible and happening today, I've seen other artists truly innovate and I've been told by a manufacturer that I did things they never considered with the newest materials ).
With that simple illustration of the difference between creativity and innovation, I'll ask you to think of them as a Venn Diagram that can overlap, but separately they still have value. A creative work and/or masterful work of art or technology can be extolled or appreciated regardless of whether it's new or innovative. An innovative work should be appreciated and differentiated because it brings something new to the world for further exploration and problem solving.
If we can learn to see the difference, then we can further appreciate the intersection of all three. We can also better understand how to foster each of them. Technology has advanced so greatly that we can easily "create" software solutions quickly today without mastery. We can do the same with paintings. For some problems, that ease of creating raw, imperfect work has it's benefit because it can allow for greater experimentation. So there's no benefit in forcing the three terms together.
I think the interwoven use of the two terms is the result of the new meaning of creativity. Let's be careful about that. Innovation is more than creativity - innovation is not only new, it's potentially disruptive and groundbreaking. It's noteworthy because of what it changes.
 |
| Song of the Angels by William-Adolphe Bouguereau |
The Innovators by Walter Isaacson
Recently published in 2015, this book is a fun history lesson about the innovators that led to the creation of modern computers. The grouping of stories illustrates how a series of inventors and innovative thinkers influenced each other and virtually collaborated (or shared ideas) and competed with each other through the ages. There is much more to the book than I can capture in a paragraph but I couldn't stop reading and hope you enjoy it too.
Innovators Dilemma by Clayton M. Christensen
Fantastic and now classic book on the challenges that prevent organizations from innovating and engaging disruptive technology. Christensen published a more recent book The Innovator's Solution.
Innovator's Hypotheseis by Michael Schrage
In response to the challenge of innovation Schrage proposes that innovative teams work in parallel groups he calls 5x5. Leveraging what I consider the power of cooperation in an iterative cycle with a feedback loop and constraints this is the first published format that resonates with my own past experience of how creativity and innovation actually *work*.
The Art of Innovation Tom Kelley and Johnathan Littman
Also a classic for those of us who are into "design thinking". IDEO as an organization are experts in the process of creative and innovative thinking.
The Best Definition of Innovation by Scott Burkun
While updating this post, I found this refreshingly confronting article by Scott Burkun imploring us to use the word sparingly and he provides his definition of innovation in order to distinguish it.
Friday, October 14, 2016
Creating a Kubernetes Cluster With kube-aws
Kubernetes is one of the most promising container orchestration frameworks, because it is portable—not tied to a particular cloud provider (even though Google created it)—and because it is full-featured.
Unfortunately, creating a Kubernetes cluster is a little bit complex. The Kubernetes project currently provides two ways to create a cluster: (1) the kube-up script, which can be accessed from https://get.k8s.io, and (2) the kops tool, which has great potential but is relatively immature at this point. You can also create a cluster by hand, if you know what you are doing. But if you are working in AWS, then the best way currently, in my opinion, is by using the CoreOS “kube-aws” tool.
What Is a Kubernetes Cluster?
A cluster is just a collection of machines that are managed by a Kubernetes cluster controller. The machines can be real machines or virtual machines (VMs). Using real machines is more efficient since one then does not have the overhead of a hypervisor, which is redundant when using containers. (It remains to be seen what will become of the large commercial hypervisor-based VM ecosystem that exists today.)
If you are a software developer in a large organization, you will likely already have a cluster to deploy to—it would have been created for you. Alternatively, some teams prefer to create a cluster for each application, or for groups of applications. Regardless, someone has to create the cluster, and if you are working on your own, or in a self-sufficient team, then you do.
As I said above, I recommend the CoreOS kube-aws tool for creating Kubernetes clusters. Instructions for obtaining and using kube-aws can be found here. However, those instructions, while excellent overall, leave a few details out. Below I provide some tips that will hopefully fill in the gaps.
AWS Steps
I generally do all of my command line work on an AWS node, and I recommend that you do that too for setting up a Kubernetes cluster in AWS. Create a Linux VM in AWS, ssh into that instance, and then perform the steps described in the kube-aws instructions, with the tips below.
Obtain kube-aws
Tip: To see the full list of options for gpg2, type,
gpg2 --dump-options
Tip: To download kube-aws using curl, use the -L and -O options:
curl -L -O https://github.com/coreos/coreos-kubernetes/releases/download/v0.8.2/kube-aws-linux-amd64.tar.gz
Configure AWS Access
Tip: The AWS command line tools will have to be installed on your system. See here if you need to install them.
Tip: Instead of creating ~/.aws/credentials and ~/.aws/config files, you can set these environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
Configure Key-Based Access to AWS Instances
Tip: Create an AWS EC2 SSH key pair, if you do not already have one. You give the key pair a name when you create it, and the public key is retained by AWS. You can then access it via the “Key Pairs” section of the EC2 console.
When a key pair is created, you will be instructed to download the public and private key pair and store the private key securely—either in your ~/.ssh directory, or in a project-specific location, and it should have 0400 access. You will specify the name of this key pair when you configure the Kubernetes cluster, and Kubernetes will use AWS to install the public key on each machine in the cluster. The presence of that key on each machine enables you to perform ssh -i <private-key-location> <userid>@<host> to connect to the machines that were created.
Tip: Create an SSL/TLS key pair. AWS provides a tool/system called KMS for this purpose. This is used for establishing SSL/TLS connections between users of your application and your Kubernetes load balancer.
Tip: In order to create the domain name that kube-aws requires as the name of the cluster controller, you will have to have an existing DNS domain so that you can create the cluster controller’s domain as a subdomain. To create that existing domain in AWS, you can use the Route 53 service: create a “Hosted Zone”, for example mydomain.com, and then for the Kubernetes cluster controller, specify a new subdomain, such as cluster.mydomain.com. kube-aws will automatically create the subdomain.
Create Cloud Formation Assets
kube-aws works by creating an AWS Cloud Formation file. Cloud Formation is AWS’s native computing environment orchestration tool. This is a good thing because by using Cloud Formation, kube-aws benefits from the maturity and robustness of Cloud Formation.
Create Directory For the Cluster Configuration
When creating a cluster, you keep the cluster’s configuration files in a directory on the machine from which you plan to administer the cluster. This is similar to other environment creation tools such as vagrant. Create a directory for your cluster:
mkdir mycluster
cd mycluster
Create Configuration File
Tip: If you want kube-aws to create the subdomain cluster.mydomain.com off of the existing domain mydomain.com, your AWS region is us-west-2, you would like the cluster to be created in zone us-west-2b of region us-west-2, your SSH key pair name is MyKeyPair (in which case, you will have downloaded a file called MyKeyPair.pem from AWS), and your SSL/TLS key (see “Encryption Keys” in the AWS “Identity and Access Management” console) ARN is arn:aws:kms:my-region:xxxxxxxxxxx:key/xxxxxxxxxxxxxxxx,
kube-aws init \
--cluster-name=mycluster \
--external-dns-name=cluster.mydomain.com \
--region=us-west-2 \
--availability-zone=us-west-2b \
--key-name=MyKeyPair \
--kms-key-arn="arn:aws:kms:my-region:xxxxxxxxxxx:key/xxxxxxxxxxxxxxxx"
Tip: The cluster-name must be unique within your AWS account.
Tip: The second level domain name of the subdomain specified by the external-dns-name—i.e., the “mydomain.com” part of cluster.mydomain.com—must be a valid registered DNS name. Merely creating a Hosted Zone in AWS for that domain will not be sufficient. You must create the hosted zone for the domain name (e.g., mydomain.com) in AWS, and also register the domain name with a domain registrar. AWS provides domain registration—see here.
Tip: Edit the resulting cluster.yaml file. E.g., set these to the values you prefer:
createRecordSet: true
hostedZoneId: <id>
workerCount: 2
workerInstanceType: m3.medium
workerRootVolumeSize: 30
tlsCADurationDays: 3650
tlsCertDurationDays: 365
where the <id> is the “Hosted Zone ID” of your AWS Route 53 hosted zone.
By default, kube-aws will create its own AWS Virtual Private Cloud (VPC). However, you can set the vpcId parameter in the cluster.yaml file if you want your cluster to use an existing VPC that you have created through other means.
Note: As of this writing, the generated cluster.yaml file uses the v1.3.6_coreos.0 version of the CoreOS image quay.io/coreos/hyperkube, which—according to CoreOS’s own scan, has 95 security vulnerabilities. You can select a different version by editing the kubernetesVersion value in the cluster.yaml file. However, even the most recent version (as of this writing) has 72 vulnerabilities. Some of the vulnerabilities involve denial of service attacks on the SSL implementation. This means that the cluster controller should only be deployed in a secure network and should not be directly accessible from the Internet—access from the Internet should only be via a proxy or a VPN/VPC.
Generate Cloud Formation
All we need to do is type,
kube-aws render
kube-aws validate
The output is a 600+ line stack-template.json file, which is a Cloud Formation file, and a kubeconfig file, which specifies which credentials to use when authenticating to your cluster.
Launch Cluster
kube-aws up
At this point, it displays a message, “Creating AWS resources. This should take around 5 minutes” - and indeed it does take that long, and gives you no feedback while it is working, but you can see progress if you go to the AWS Cloud Formation console, where you should see the “stack” listed with a “CREATE_IN_PROGRESS” message. When it finished the status should change to “CREATE_COMPLETE”.
When it completes you should also have command line output like this:
Creating AWS resources. This should take around 5 minutes.
Success! Your AWS resources have been created:
Cluster Name: mycluster
Controller IP: 100.100.100.100
The containers that power your cluster are now being downloaded.
You should be able to access the Kubernetes API once the containers finish downloading.
If you then go to your AWS EC2 instance console, you should now see something like what is shown in the Figure below. Notice the first two instances: these are what I got when I launched a cluster for ScaledMarkets’ safeharbor service. If I had set the workerCount to be more than one, then there would be that many instances of kube-aws-worker.
Verify that the Cluster Is Accessible
You can now perform Kubernetes commands to manage your cluster. The primary command is the kubectl command. I launched my cluster on an x86-64 Centos 7 system, so I put the /kubernetes/platforms/linux/amd64 directory in my path. To verify access to the cluster, try
kube-aws status
That should give you something like this:
Cluster Name: mycluster
Controller IP: 100.100.100.100
kubectl --kubeconfig=kubeconfig get nodes
Assuming that your DNS domain is registered and that there is an AWS Hosted Zone for it, you should get something like this:
NAME STATUS AGE
ip-10-0-0-156.us-west-2.compute.internal Ready 2m
ip-10-0-0-50.us-west-2.compute.internal Ready,SchedulingDisabled 2m
Deleting a Cluster
The command,
kube-aws destroy
deletes the cluster. However, it does not give you any feedback on progress. If you want to see that, go to your AWS Cloud Formation console, and you should see the Cloud Formation “stack” created by kube-aws with the message, DELETE_IN_PROGRESS.
Unfortunately, kube-aws destroy does not always work: it often fails to remove things like the VPC, load balancer, and other things that kube-aws up created. Thus, you might have to go into the Cloud Formation console event list to look at the messages and see which deletions failed, then then delete those things manually.
Using the Kubernetes Dashboard
Kubernetes has a nice dashboard for viewing and managing your cluster. However, your cluster is in AWS. If you are SSH-ing into a VM in AWS as I recommended, you will not be able to launch a browser there to view the dashboard. There are many ways to get around that. One is to boot a VM with X-Windows or Wayland and access that remotely. I have not tried that, so I can’t give you advice there. Another approach is to launch the AWS-based cluster from your local machine instead of from a VM in AWS as I have recommended. I don’t like to execute projects on my local machine because then I muck up my local machine—I like to always work from a VM that I can then blow away, using a separate VM for each project—but you can also avoid mucking up your local machine by creating a local VM with a console (“headed”) and working from that. It should work fine and be secure, but I don’t work that way. Finally, you can use run the Kubernetes proxy in AWS and access the proxy via the VM’s public IP address. To do that, type this on your AWS VM command line:
kubectl --kubeconfig kubeconfig proxy --accept-hosts="^*$" --address="<your-AWS-VM-private-IP-address>"
where <your-AWS-VM-private-IP-address> is the private IP address of your AWS VM. You can obtain that from the EC2 instance console. Then add port 8001 to the security group for your AWS VM, allowing any IP address for egress, but limiting to your local IP address for ingress—that is the “Source” setting in the AWS Security Group.
You can then view the dashboard at,
http://<your-AWS-VM-public-IP-address>:8001/ui
where <your-AWS-VM-public-IP-address> is the public IP address of your AWS VM.
The dashboard should look something like what is shown below.
You can click on the various dashboard elements to drill down—you can even edit a pod’s configuration and restart the pod, but I have not tried that. If you click on a pod, you will then see a detail page, showing the pod’s full name—the name shown on the dashboard main page is only an abbreviated name. Importantly, you can use the dashboard to find out the generated names of your pods, so that you can then perform actions such as attaching to a container:
kubectl --kubeconfig=kubeconfig attach <pod> -c <container>
Note that the kubectl proxy command blocks—canceling it with ^C will terminate the proxy—so you will probably want to run it with something like,
nohup kubectl --kubeconfig kubeconfig proxy --accept-hosts="^*$" --address="<your-AWS-VM-private-IP-address>" > log.out 2> log.err < /dev/null &
Note: Using the proxy is not a secure setup, because while you have restricted dashboard access to your IP address, IP addresses can be spoofed, and the proxy uses HTTP (not HTTPS). Therefore, you should only use the proxy from within a VPC/VPN or secure local network. However, when working in AWS I strongly recommend always working within a VPC/VPN anyway.
Conclusion
Once a cluster has been defined, we have one command to launch an entire cluster of machines, and we can administer that cluster with the kubectl command. The hard part was in creating the AWS keys and config files, but now that that has been done, we can create additional clusters with ease. Thus, we now have a repeatable and reliable process for standing up Kubernetes clusters—using AWS’s native features.
Now what? We have a cluster, but what can we do with it? That will be the topic of the next article, “Deploying a Kubernetes Service In Three Easy Steps”, in which I explain how to define and deploy an application configuration with Kubernetes.
Subscribe to:
Posts (Atom)