Matt Dorn

oc cluster up vs. minishift vs. CKD vs. Ansible Playbooks

Matt Dorn — Mon, 01 Oct 2018 12:00:00 -0500

As you may already know, OpenShift Origin, the Red Hat distribution of Kubernetes, was recently renamed OKD.

I couldn't find a document that explained the difference between the various methods for deploying an OKD dev environment so I thought this would be helpful.

oc cluster up

oc cluster up is a command that is part of the oc (OpenShift Client) binary. Linux is currently the only supported platform for running the command. It will call Docker on the host where the command is run and bootstrap a local OKD cluster. The version of OKD that gets created is relative to your version of oc. It was first introduced in OpenShift Origin 1.3 and was created to give engineers a quick way to test their work. It is best suited for creating quick, disposable environments that you don't plan on persisting (i.e. testing, adding to your CI/CD workflow, etc.). If you are eager to try out the latest OKD release, you'll usually be able to get your hands on it via oc cluster up.

You can download the oc binary from here and place it in your path.

You can check out all the available oc cluster commands by running oc cluster --help.

More info can be found here as well.

minishift

minishift is supported on a variety of platforms including MacOS, Windows, and Linux. It uses libmachine for provisoning a single-node OKD cluster and is compatible with a variety of hypervisors like VirtualBox, VMware Fusion, xhyve, hyperkit, etc). Behind the scenes, it is actually using oc cluster up. It has some very cool profiling features for creating and persisting a variety of dev environments. It's great if you are looking for a local, reusable, dev environment that's isolated in a virtual machine.

More info on installing Minishift can be found here.

CDK (Container Development Kit)

CDK could best be described as an Enterprise version of minishift. It is a downstream productized version of minishift that allows a user with a Red Hat developer subscription to deploy OpenShift on Red Hat Enterprise Linux (RHEL).

More info can be found here.

Installing manually via Ansible Playbooks

If you want to deploy an environment the "official way", you can always provision a single virtual machine and manually run the ansible-openshift playbooks.

Check out this video from Grant Shipley that shows you how to do it with a handy script.

Certified Kubernetes Application Developer Exam

Matt Dorn — Tue, 15 May 2018 12:00:00 -0500

In case you didn't notice, the Cloud Native Computing Foundation recently dropped another Kubernetes exam called the Certified Kubernetes Application Developer (CKAD).

I took the exam a few weeks ago and thought it was pretty good. Unlike the Certified Kubernetes Administrator Exam (CKA), the CKAD only focuses on the Developer persona. This makes the CKAD much easier than the CKA but keep in mind you only have two hours to complete all tasks.

Note

The CNCF has updated their open-book policy for both CKA and CKAD. You are only permitted to open one additional web browser tab to access official documentation at kubernetes.io during the exam. No other websites are permitted.

Objectives

The objectives for the CKAD can be found throughout the Official Certified Kubernetes Application Developer Exam Curriculum.

Together, all of these domains make up 100% of the exam.

Core Concepts - 13%

Understand Kubernetes API primitives
Create and configure basic Pods

Multi-Container Pods - 10%

Understand Multi-Container Pod design patterns (e.g. ambassador, adapter, sidecare)

Pod Design - 20%

Understand how to use Labels, Selectors, and Annotations
Understand Deployments and how to perform rolling updates
Understand Deployments and how to perform rollbacks
Understand Jobs and CronJobs

State Persistence - 8%

Understand PersistentVolumeClaims for storage

Configuration - 18%

Understand ConfigMaps
Understand SecurityContexts
Define an application’s resource requirements
Create and consume Secrets
Understand ServiceAccounts

Observability - 18%

Understand LivenessProbes and ReadinessProbes
Understand container logging
Understand how to monitor applications in Kubernetes
Understand debugging in Kubernetes

Services & Networking - 13%

Understand Services
Demonstrate basic understanding of NetworkPolicies

About the Exam Environment

The exam environment is the exact same as the CKA and described in my last post.

Good Luck!

Have fun preparing and taking the Certified Kubernetes Developer Exam! Check out the CNCF's FAQ for additional info.

Certified Kubernetes Administrator Exam

Matt Dorn — Sun, 10 Sep 2017 12:00:00 -0500

The Cloud Native Computing Foundation has officially released the brand new Certified Kubernetes Exam (CKA).

I took the exam last week and loved it! It was extremely challenging and covered many different aspects of Kubernetes.

Exam Personas

The exam can best be described as a series of objectives that cover two target personas: Developers and Operators.

Developers

The developer section requires you to understand how to directly interact with the Kubernetes API via kubectl.

These objectives require you to act as the developer deploying cloud-native applications on Kubernetes by creating Pods, Jobs, Deployments, Services and ConfigMaps. You should understand how to run ad-hoc kubectl commands as well as create object manifest files in JSON or YAML format.

This makes up about 60% of the exam.

Operators

The operators portion of the exam requires you to put on your System Engineer hat and understand how to install, manage, and troubleshoot live Kubernetes clusters! You must be familiar with all aspects of the Kubernetes installation process, including flags and settings for the kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, kube-proxy, and etcd.

This makes up about 40% of the exam.

Objectives

The objectives for the Developer/Operator personas can be found throughout the Official Certified Kubernetes Administrator Exam Curriculum.

Together, all of these domains make up 100% of the exam.

Core Concepts - 19%

Understand the Kubernetes API primitives.
Understand the Kubernetes cluster architecture.
Understand Services and other network primitives.

Security - 12%

Know how to configure authentication and authorization.
Understand Kubernetes security primitives.
Know how to configure network policies.
Create and manage TLS certificates for cluster components.
Work with images securely.
Define security contexts.
Secure persistent key value store.

Installation, Configuration, Validation - 12%

Design a Kubernetes cluster.
Install Kubernetes masters and nodes.
Configure secure cluster communications.
Configure a HA Kubernetes cluster.
Know where to get the Kubernetes release binaries.
Provision underlying infrastructure to deploy a Kubernetes cluster.
Choose a network solution.
Choose your Kubernetes infrastructure configuration.
Run end-to-end tests on your cluster.
Analyze end-to-end test results.
Run Node end-to-end tests.

Networking - 11%

Understand the networking configuration on the cluster nodes.
Understand Pod networking concepts.
Understand service networking.
Deploy and configure network load balancer.
Know how to use Ingress rules.
Configure secure cluster communications.
Configure a HA Kubernetes cluster.
Know where to get the Kubernetes release binaries.
Provision underlying infrastructure to deploy a Kubernetes cluster.
Choose a network solution.
Choose your Kubernetes infrastructure configuration.
Run end-to-end tests on your cluster.
Analyze end-to-end test results.
Run Node end-to-end tests.

Cluster Maintenance - 11%

Understand Kubernetes cluster upgrade process.
Facilitate operating system upgrades.
Implement backup and restore methodologies.

Troubleshooting - 10%

Troubleshoot application failure.
Troubleshoot control panel plane failure.
Troubleshoot worker node failure.
Troubleshoot networking.

Application Lifecycle Management - 8%

Understand Deployments and how to perform rolling updates and rollbacks.
Know various ways to configure applications.
Know how to scale applications.
Understand the primitives necessary to create a self-healing application.

Storage - 7%

Understand Persistent Volumes and know how to create them.
Understand access modes for volumes.
Understand Persistent Volume Claims.
Understand Kubernetes storage objects.
Know how to configure applications with Persistent Storage.

Scheduling - 5%

Use label selectors to schedule Pods.
Understand the role of DaemonSets.
Understand how Resource Limits can affect Pod scheduling.
Understand how to run multiple schedulers and how to configure Pods to use them.
Manually schedule a pod without a scheduler.
Display scheduler events.
Know how to configure the Kubernetes scheduler.

Logging/Monitoring - 5%

Understand how to monitor all cluster components.
Understand how to monitor applications.
Manage cluster component logs.
Manage application logs.

About the Exam Environment

The exam expects its test-takers to be proficient in completing all objectives in a terminal environment.

Here is a visual representation of the exam console as described in the Certification and Confidentiality Agreement:

The exam console is embedded into the browser. It is composed of two primary parts: The Content Panel and the Terminal Panel.

The Content Panel is the section that displays the exam timer and objectives. You can use the next and back button to move to each objective.

The Terminal Panel gives you full access to an Ubuntu 16.04 environment with previously installed programs including kubectl and SSH to access a Kubernetes cluster environment.

It's Open Book!

Although you can't use your personal notes for the exam, you can use the net to browse official Kubernetes documentation.

Good Luck!

Have fun preparing and taking the Certified Kubernetes Administrator!

Kubernetes Non-Masquerade CIDR

Matt Dorn — Thu, 27 Jul 2017 11:19:00 -0500

I've noticed a few people running into DNS issues when attempting to install Kubernetes from scratch on VirtualBox with Vagrant.

As you may already know, VirtualBox's default NAT network assigns an IP address from a 10.0.2.0/24 CIDR to any virtual machine with a network adapter set to NAT. This will result in your virtual machine receiving a DHCP option that sets /etc/resolv.conf to a nameserver that is typically at something like 10.0.2.3.

From Ubuntu 16.04:

ubuntu@worker1:~$ cat /etc/resolv.conf
 # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.2.3

For node-to-node communication, we can use a VirtualBox host-only network (For example, 192.168.56.0/24) and statically assign an ip address to eth1 on our virtual machines.

This will result in the interfaces on our virtual machines looking something like this:

ubuntu@worker1:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
     valid_lft forever preferred_lft forever
  inet6 ::1/128 scope host
     valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
  link/ether 08:00:27:82:73:25 brd ff:ff:ff:ff:ff:ff
  inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
     valid_lft 86041sec preferred_lft 86041sec
  inet6 fe80::a00:27ff:fe82:7325/64 scope link
     valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
  link/ether 08:00:27:fa:05:a3 brd ff:ff:ff:ff:ff:ff
  inet 192.168.56.56/24 brd 192.168.56.255 scope global enp0s8
     valid_lft forever preferred_lft forever
  inet6 fe80::a00:27ff:fefa:5a3/64 scope link
     valid_lft forever preferred_lft forever

To keep things simple in a typical Kubernetes environment, you may want to have each worker node's pod network on a unique /24. This would result in each worker node's network bridge getting assigned something like 10.200.0.1/24, 10.200.1.1/24, 10.200.2.1/24, and so on. If using kubenet, the out-of-the-box Kubernetes networking plugin, this can be achieved by setting the --cluster-cidr flag on the kube-controller-manager daemon to 10.200.0.0/16.

Here's an example of one commonly used kube-controller-manager systemd unit config from Kelsey Hightower's Learn Kubernetes the Hard Way

cat > kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/bin/kube-controller-manager \\
  --address=0.0.0.0 \\
  --allocate-node-cidrs=true \\
  --cluster-cidr=10.200.0.0/16 \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file="/var/lib/kubernetes/ca.pem" \\
  --cluster-signing-key-file="/var/lib/kubernetes/ca-key.pem" \\
  --leader-elect=true \\
  --master=http://${INTERNAL_IP}:8080 \\
  --root-ca-file=/var/lib/kubernetes/ca.pem \\
  --service-account-private-key-file=/var/lib/kubernetes/ca-key.pem \\
  --service-cluster-ip-range=10.32.0.0/16 \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

After creating your DNS deployment in the kube-system namespace, resolution of service names inside the cluster may work fine:

ubuntu@worker1:~$ kubectl exec -it mypod /bin/sh
/ # nslookup kubernetes
Server:    10.32.0.10
Address 1: 10.32.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.32.0.1 kubernetes.default.svc.cluster.local

But resolution to something outside your cluster doesn't work:

/ # nslookup kubernetes.io
Server:    10.32.0.10
Address 1: 10.32.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes.io'

If we check out the upstream kube-dns deployment object manifest the dnsmasq container does not have the --resolv-file flag set. This flag would allow us to specify our own customized resolv.conf. By default, the dnsmasq container will use the /etc/resolv.conf of the node the pod resides on for resolving all non-cluster queries. In our case with VirtualBox NAT network, it will use the 10.0.2.3 nameserver.

Rather than play around with setting a new DNS server like 8.8.8.8, let's do some tcpdumping on the worker node's NAT network interface to see if we can see the packets attempting to hit the VirtualBox DNS:

ubuntu@worker1:~$ sudo tcpdump -n "dst host 10.0.2.3"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:29:54.364528 IP 10.200.0.3.15612 > 10.0.2.3.53: 28501+ AAAA? kubernetes.io. (31)
02:29:54.364629 IP 10.200.0.3.15612 > 10.0.2.3.53: 28501+ AAAA? kubernetes.io. (31)
02:29:59.370219 IP 10.200.0.3.22452 > 10.0.2.3.53: 53929+ AAAA? kubernetes.io. (31)
02:29:59.370413 IP 10.200.0.3.22452 > 10.0.2.3.53: 53929+ AAAA? kubernetes.io. (31)

It looks like the source IP address is still the original pod IP.

Checking iptables NAT table, we can see the following in the POSTROUTING chain:

ubuntu@worker1:~$ sudo iptables -t nat -vxnL POSTROUTING

Chain POSTROUTING (policy ACCEPT 8 packets, 480 bytes)
    pkts      bytes target     prot opt in     out     source               destination
      861    57670 KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
      49     2940 MASQUERADE  all  --  *      *       0.0.0.0/0           !10.0.0.0/8           /* kubenet: SNAT for outbound traffic from cluster */ ADDRTYPE match dst-type !LOCAL

It looks like all traffic destined for something within 10.0.0.0/8 will not be source natted!

Where did it get this 10.0.0.0/8 network? As you can see here, kubelet automatically sets this cidr value when the --non-masquerade-cidr flag for kubelet is not specified. The intention is to ensure proper pod-to-pod communication within the cluster by preserving the pod's source IP. In my dev environment, I have manually generated host-routes, but you may be using flannel or another pod networking solution. Anyway, it looks like this will be a problem since our VirtualBox DNS falls into the 10.0.0.0/8 RFC 1918 range.

We could certainly work around this by changing our cluster-cidr to something like 192.168.0.0/16. We could also manually modify iptables rules on all of our worker nodes. Let's try something easier.

The ip-masq-agent

Enter the ip-masq-agent!

In Kubernetes 1.7 there is some new code that allows you to completely void out the creation of the --non-masquerade-cidr by setting its value to 0.0.0.0/0, i.e. --non-masquerade-cidr=0.0.0.0/0.

I will create ConfigMap that says I do not want to SNAT any traffic destined for the cluster, in my case the user-provided cluster-cidr and service-cluster-ip ranges: 10.200.0.0/16 and 10.32.0.0/16. This is pretty cool because the kubelet's --non-masquerade-cidr is limited to one CIDR. The template for the ConfigMap is available here. Here is my ConfigMap:

nonMasqueradeCIDRs:
  - 10.200.0.0/16
  - 10.32.0.0/16
masqLinkLocal: false
resyncInterval: 60s

masqLinkLocal determinws whether to masquerade traffic to 169.254.0.0/16. False by default.
resyncInterval sets the interval at which the agent attempts to reload config from disk.

Deploy the ip-masq-agent DaemonSet

kubectl create -f https://raw.githubusercontent.com/kubernetes-incubator/ip-masq-agent/v2.0.1/ip-masq-agent.yaml

We should see a brand new POSTROUTING rule that points to a new chain called IP-MASQ-AGENT:

ubuntu@worker1:~$ sudo iptables -t nat -vxnL POSTROUTING
Chain POSTROUTING (policy ACCEPT 4 packets, 240 bytes)
  pkts      bytes target     prot opt in     out     source               destination
 11235   674191 KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
 10067   604182 IP-MASQ-AGENT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ip-masq-agent: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom IP-MASQ-AGENT chain */ ADDRTYPE match dst-type !LOCAL

Let's look at the new chain:

ubuntu@worker1:~$ sudo iptables -t nat -vxnL IP-MASQ-AGENT
Chain IP-MASQ-AGENT (1 references)
    pkts      bytes target     prot opt in     out     source               destination
       0        0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
      12      720 RETURN     all  --  *      *       0.0.0.0/0            10.200.0.0/16        /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
       0        0 RETURN     all  --  *      *       0.0.0.0/0            10.32.0.0/16         /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
       0        0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL

Our non-masquerade rules have been applied!

Let's check DNS resolution:

ubuntu@worker1:~$ kubectl exec -it mypod /bin/sh
/ # nslookup kubernetes.io
Server:    10.32.0.10
Address 1: 10.32.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.io
Address 1: 23.236.58.218 218.58.236.23.bc.googleusercontent.com
/ #

It works!

Let's check out tcpdump again:

ubuntu@worker1:~$ sudo tcpdump -i enp0s3 dst host 10.0.2.3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes
12:30:35.522668 IP 10.0.2.15.65039 > 10.0.2.3.domain: 36319+ AAAA? google.com. (28)
12:30:35.522781 IP 10.0.2.15.65039 > 10.0.2.3.domain: 36319+ AAAA? google.com. (28)
12:30:35.522928 IP 10.0.2.15.63090 > 10.0.2.3.domain: 3284+ AAAA? google.com. (28)
12:30:35.523002 IP 10.0.2.15 > 10.0.2.3: ICMP 10.0.2.15 udp port 65039 unreachable, length 64
12:30:35.523099 IP 10.0.2.15.12876 > 10.0.2.3.domain: 44566+ AAAA? google.com. (28)
12:30:35.523476 IP 10.0.2.15.57223 > 10.0.2.3.domain: 37209+ PTR? 3.2.0.10.in-addr.arpa. (39)
12:30:35.523639 IP 10.0.2.15.40473 > 10.0.2.3.domain: 10359+ AAAA? google.com. (28)

Looks like SNAT is working for our request to the VirtualBox DNS.

If you are interested in diving more into customizing upstream dns servers and/or private dns zones, check out this recent kubernetes blog article.

Certified OpenStack Administrator Exam Book

Matt Dorn — Wed, 05 Apr 2017 12:00:00 -0500

My new book, Preparing for the Certified OpenStack Administrator Exam , will be out in July 2017 on Packt Publishing.

The exam prepares students for the upcoming Newton version of the exam on Ubuntu 16.04. The exam currently tests on Liberty but will be upgraded to Newton in May 2017.

Here is an excerpt from the first chapter:

Benefits of Passing the Exam

Ask anyone about getting started in the IT world and they may suggest looking into industry-recognized technical certifications. IT certifications measure competency in a number of areas and are a great way to open doors to opportunities. While they certainly should not be the only determining factor in the hiring process, achieving them can be a measure of your competence and commitment to facing challenges.

If You Pass...

Upon completion of a passing grade, you will receive your certificate. Laminate, frame, or pin it to your home office wall or work cubicle! It’s proof that you have met all the requirements to become an official OpenStack Administrator. The certification is valid for three years from the pass date so don't forget to renew!

In addition to the certification, a COA badge will appear next to your name in the OpenStack Foundation Member Directory.

Note

The OpenStack Foundation has put together a great tool for helping employers verify the validity of COA certifications. Check out the Certified OpenStack Administrator Verification Tool.

7 Steps to Becoming a Certified OpenStack Administrator!

Let's begin by walking through some steps to become a Certified OpenStack Administrator!

Step 1 - Study!

Practice. Practice. Practice. Use this book and the included OpenStack All-in-One Virtual Appliance as a resource as you begin your Certified OpenStack Administrator journey. If you still find yourself struggling with the concepts and objectives in this book, you can always refer to the Official OpenStack Documentation or even seek out a live training class at the OpenStack Training Marketplace.

Step 2 - Purchase!

Once you feel that you’re ready to conquer the exam, head to the Official Certified OpenStack Administrator Homepage and click on Get Started. After signing in, you will be directed to checkout to purchase your exam. The OpenStack Foundation accepts all major credit cards and as of April 2017, costs $300.00 USD but is subject to change so keep an eye out on the website. You can also get a FREE retake within 12 months of the original exam purchase date if you do not pass on the first attempt.

Note

To encourage academia students to get their feet wet with OpenStack technologies, the OpenStack Foundation is offering the exam for $150.00 (50% off the retail price) with a valid student ID. Check out https://www.openstack.org/coa/student/ for more info.

Step 3 - COA Portal Page

Once your order is processed, you will receive an email with access to the COA Portal. Think of the portal as your personal COA website where you can download your exam receipt and keep track of your certification efforts. Once you take the exam, you can come back to the COA Portal to check your exam status, exam score, and even download certificates and badges for displaying on business cards or websites!

Step 4 - Hardware Compatibility Check

The COA exam can be taken from your personal laptop or desktop but you must ensure that your system meets the exam’s minimum system requirements. A link on the COA Portal page will present you with the Compatibility Check Tool which will run a series of tests to ensure you meet the requirements. It will also assist you in downloading a Chrome plugin for taking the exam. At this time, you must use the Chrome or Chromium browser and have access to reliable internet, a webcam, and microphone. Here is a current list of requirements:

Step 5 - Identification

You must be at least 18 years old and have proper identification to take the exam!

Any of the following pieces of identification are acceptable:

Passport

Government-issued driver's license or permit

National identity card

State or province-issued identity card

Step 6 - Schedule the Exam

I personally recommend scheduling your exam a few months ahead of time to give yourself a realistic goal. Click on the Schedule Exam link on the COA Portal to be directed and automatically logged into the Exam Proctor Partner website. Once logged into the site, type OpenStack Foundation in the search box and select the COA exam. You will then choose from available dates and times. The latest possible exam date you can schedule will be 30 days out from the current date. Once you have scheduled it, you can cancel or reschedule up to 24 hours before the start time of the exam.

Step 7 - Take the Exam!

Your day has arrived! You've used this book and have practiced day and night to master all of the covered objectives! It's finally time to take the exam!

One of the most important factors determining your success on the exam is the location. You cannot be in a crowded place! This means no coffee shops, work desks, or football games! The testing location policy is very strict, so please consider taking the exam from home or perhaps a private room in the office.

Log into the COA Portal fifteen minutes before your scheduled exam time. You should now see a Take Exam link which will connect to the Exam Proctor Partner website so you can connect to the testing environment.

Once in the exam environment, an Exam Proctor chat window will appear and assist you with starting your exam. You must allow sharing of your entire operating system screen (this includes all applications), webcam, and microphone. t’s time to begin! You have two and a half hours to complete all exam objectives. You're almost on your way to becoming a Certified OpenStack Administrator!

About the Exam Environment

The exam expects its test-takers to be proficient in interacting with OpenStack via the Horizon dashboard and command-line interface. Here is a visual representation of the exam console as outlined in the COA Candidate Handbook:

The exam console is embedded into the browser. It is composed of two primary parts: The Content Panel and the Dashboard/Terminal Panel.

The Content Panel is the section that displays the exam timer and objectives. As per the COA Handbook, exam objectives can only be navigated linearly. You can use the next and back button to move to each objective.

The Dashboard/Terminal Panel gives you full access to an OpenStack environment. Chapter 2 of this book will assist you with getting your practice OpenStack environment environment up and running so you can work through all the objectives.

Note

The exam console terminal is embedded in a browser and you cannot SCP (secure copy) to it from your local system. Within the terminal environment, you are permitted to install a multiplexor such as screen, tmux, or byobu if you think these will assist you but are not necessary for successful completion of all objectives.

You are not permitted to browse websites, email, or notes during the exam but you are free to access the Official OpenStack Documentation. This can be a major waste of time on the exam and shouldn’t be necessary after working through the exam objectives in this book. You can also easily copy and paste from the objective window into the Horizon dashboard or terminal.

If you struggle with a question, move on! Hit the next button and try the next objective. You can always come back and tackle it before time is up.

The exam is scored automatically within 24 hours and you should receive the results via email within 72 hours after exam completion. At this time, the results will be made available on the COA Portal. Please review the Professional Code of Conduct on the OpenStack Foundation Certification Handbook.

The Exam Objectives

Let's now take a look at the objectives you will be responsible for performing on the exam. As of March 2017, these are all the exam objectives published on the Official COA website. These objectives will test your competence and proficiency in the domains listed below. These domains cover multiple core OpenStack services as well as general OpenStack troubleshooting. Together, all of these domains make up 100% of the exam.

Note

Because some of the objectives on the official COA Requirements list overlap, this book utilizes its own convenient strategy to ensure you can fulfill all objectives within all content areas.

Getting To know OpenStack - 3% - Chapter 1

Understand the components that make up the cloud
Use the OpenStack API/CLI

Keystone - Identity management - 12% - Chapter 3

Manage Keystone catalogue services and endpoints
Manage/Create domains, groups, projects, users, and roles
Create roles for the environment
Manage the identity service
Verify operation of the Identity service

Glance - Image management - 10% - Chapter 4

Deploy a new image to an OpenStack instance
Manage image types and backends
Manage images (e.g. add, update, remove)
Verify operation of the Image Service

Nova - Compute - 15% - Chapter 5

Manage flavors
Manage compute instance actions (e.g. launch, shutdown, terminate)
Manage Nova user keypairs
Launch a new instance
Shutdown an Instance
Terminate Instance
Configure an Instance with a Floating IP address
Manage project security group rules
Assign security group to Instance
Assign floating IP address to Instance
Detach floating IP address from Instance
Manage Nova host consoles (rdp, spice, tty)
Access an Instance using a keypair
Manage instance snapshots
Manage Nova compute servers
Manage quotas
Get Nova stats (hosts, services, tenants
Verify operation of the Compute service

Neutron - Networking - 16% - Chapter 6

Manage network resources (e.g., routers, subnets)
Create external networks
Create project networks
Create project routers
Manage network services for a virtual environment
Manage project security group rules
Manage quotas
Verify operation of network service
Manage network interfaces on compute instances
Troubleshoot network issues for a tenant network (enter namespace, run tcpdump, etc)

Cinder - Block Storage - 10% - Chapter 7

Manage volume
Create volume group for block storage
Create a new Block Storage Volume and mount it to a Nova Instance
Manage quotas
Manage volumes quotas
Manage volumes backups
Backup and restore volumes
Manage volume snapshots (e.g, take, list, recover)
Verify that block storage can perform snapshotting function
Snapshot volume
Manage volumes encryption
Set up storage pools
Monitor reserve capacity of block storage devices
Analyze discrepancies in reported volume sizes

Swift - Object Storage - 10% - Chapter 8

Manage access to object storage
Manage expiring objects
Manage storage policies
Monitor space available for object store
Verify operation of Object Storage
Manage permissions on a container in object storage

Heat - Orchestration - 8% - Chapter 9

Launch a stack using a Heat/Orchestration template (e.g., storage, network, and compute
Use Heat/Orchestration CLI and Dashboard
Verify Heat/Orchestration stack is working
Verify operation of Heat/Orchestration
Create a Heat/Orchestration template that matches a specific scenario
Update a stack
Obtain detailed information about a stack

Horizon - Dashboard - 3% - Chapters 3 through 9

Verify operation of the Dashboard

Troubleshooting - 13% - Chapter 10

Analyze log files
Backup the database(s) used by an OpenStack instance
Centralize and analyze logs (e.g.,/var/log/COMPONENT_NAME, Database Server, Messaging Server, Web Server, syslog)
Analyze database servers
Analyze Host/Guest OS and Instance status
Analyze messaging servers
Analyze metadata servers
Analyze network status (physical & virtual)
Analyze storage status (local, block & object)
Manage OpenStack Services
Diagnose service incidents
Digest OpenStack environment (Controller, Compute, Storage and Network nodes)
Direct logging files through centralized logging system
Backup and restore an OpenStack instance
Troubleshoot network performance

Configure Cloud-Init to Use Admin Pass

Matt Dorn — Wed, 04 May 2016 13:59:00 -0500

When one spins up a cloud image with Cloud-Init installed, password authentication via SSH is typically disabled by default. This requires that one SSH into the instance by injecting a public key. One can override this behavior by passing the following cloud-config directives which correspond to the cc_set_passwords.py module set to run during Cloud-Init's config stage (/etc/cloud/cloud.cfg).

ssh_pwauth - edits sshd config to either allow or unallow password auth via ssh.
password - set password for default user (default user specified in /etc/cloud/cloud.cfg, i.e. ubuntu on ubuntu cloud image, centos on centos7 cloud image, etc.)
chpasswd - allows you to provide a list of user password changes as well as ensure the passwords do not expire. If expire is not set to False, one will be prompted to set a new password after authenticating. Provide RANDOM or R as the password to have it auto-generate a password. The password will appear in the console-log (nova console-log <your_instance_id>) and inside /var/log/cloud-init-output.log.

#cloud-config
ssh_pwauth: True
password: passw0rd
chpasswd:
  list: |
    user1:password1
    user2:password2
    user3:RANDOM
  expire: False

You can modify the cc_set_passwords.py module to allow it to set root's password to the metadata admin_pass value found here:

+--------------------------------------+---------------------------------------------------+
| Property                             | Value                                             |
+--------------------------------------+---------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                            |
| OS-EXT-AZ:availability_zone          |                                                   |
| OS-EXT-SRV-ATTR:host                 | -                                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                 |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000000e                                 |
| OS-EXT-STS:power_state               | 0                                                 |
| OS-EXT-STS:task_state                | scheduling                                        |
| OS-EXT-STS:vm_state                  | building                                          |
| OS-SRV-USG:launched_at               | -                                                 |
| OS-SRV-USG:terminated_at             | -                                                 |
| accessIPv4                           |                                                   |
| accessIPv6                           |                                                   |
| adminPass                            | AcSVqg3koaeS                                      |
| config_drive                         |                                                   |
| created                              | 2016-05-04T01:05:46Z                              |
| flavor                               | m1.summit (8)                                     |
| hostId                               |                                                   |
| id                                   | ed7b97ef-cea9-4140-8cd0-d30d6abba802              |
| image                                | ubuntu1604 (ad673fbe-2402-462b-b29c-d10d49252310) |
| key_name                             | -                                                 |
| metadata                             | {}                                                |
| name                                 | myinstance                                        |
| os-extended-volumes:volumes_attached | []                                                |
| progress                             | 0                                                 |
| security_groups                      | default                                           |
| status                               | BUILD                                             |
| tenant_id                            | 9d119a1e9de4498da818abe32124eb32                  |
| updated                              | 2016-05-04T01:05:46Z                              |
| user_id                              | 3545fc68adb349828d3f98893fb0d47f                  |
+--------------------------------------+---------------------------------------------------+

You can also force a specific admin pass while booting:

nova boot --image ubuntu1604 --flavor m1.summit --admin-pass mypassword mycustomrootpasswordinstance

Here is a link to a modified cc_set_passwords to fetch admin_pass from metadata and set as root password.

Cloud-Init python modules:

Ubuntu 14.04: /usr/lib/python2.7/dist-packages/cloudinit/config/

Ubuntu 16.04: /usr/lib/python3/dist-packages/cloudinit/config/

Cloud-Init Stages in Upstart and Systemd

Matt Dorn — Tue, 03 May 2016 18:46:00 -0500

Cloud-init can be found on most cloud images provided by an elastic cloud vendor. If you are rolling/baking your own cloud image, you can typically install cloud-init via your distro's package manager.

Cloud-init executes in several key stages.

Stage-1 - Init Local

/etc/init/cloud-init-local.conf (Upstart)
- start on mounted MOUNTPOINT=/
/lib/systemd/system/cloud-init-local.service (Systemd)
- Before=shutdown.target multi-user.target cloud-config.target cloud-init.service
- After=local-fs.target systemd-journald.socket basic.target system.slice
Execute: /usr/bin/cloud-init init --local
Runs prior to network setup. It looks for a NoCloud data source at /var/lib/cloud/seed/nocloud/{meta-data,user-data,network-data,vendor-data}. If present, it will copy to /var/lib/cloud/instances/<instance-id> and execute all Python modules listed in init stage at /etc/cloud/cloud.cfg.
If a data source is not found at the above, it will look for VFAT or ISO device with LABEL="cidata". If present, it will copy to /var/lib/cloud/instances/<instance-id> and excute all Python modules listed in init stage at /etc/cloud/cloud.cfg.

Stage-2 - Init

/etc/init/cloud-init.conf (Upstart)
- start on mounted MOUNTPOINT=/
- stopped cloud-init-nonet (After network is up)
/lib/systemd/system/cloud-init.service (Systemd)
- Before=multi-user.target shutdown.target systemd-user-sessions.service cloud-config.target sshd-keygen.service ssh.service
- After=network-online.target basic.target cloud-init-local.service system.slice systemd-journald.socket local-fs.target
Execute: /usr/bin/cloud-init init
datasource_list: [ NoCloud, ConfigDrive, OpenNebula, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, Ec2, CloudStack, None ] (/etc/cloud/cloud.cfg.d/90_dpkg.cfg)
By default, first datasource listed is the priority. If metadata is discovered, it is copied to /var/lib/cloud/instances/<instance-id>/.
All Python modules listed in init stage (/etc/cloud/cloud.cfg) will execute.

Stage-3 - Config

/etc/init/cloud-config.conf (Upstart)
- start on (filesystem and started rsyslog)
/lib/systemd/system/cloud-config.service (Systemd)
- Before=cloud-final.service multi-user.target shutdown.target
- After=systemd-journald.socket network-online.target syslog.target system.slice basic.target cloud-config.target
Execute: /usr/bin/cloud-init modules --mode=config
All Python modules listed in config stage (/etc/cloud/cloud.cfg) will execute.

Stage-4 - Final

/etc/init/cloud-final.conf (Upstart)
- start on (stopped rc RUNLEVEL=[2345] (starts after rc.local)
- stopped cloud-config
/lib/systemd/system/cloud-final.service (Systemd)
- Before=multi-user.target shutdown.target
- After=rc-local.service network-online.target cloud-config.service basic.target system.slice systemd-journald.socket syslog.target
Execute: /usr/bin/cloud-init modules --mode=final
All Python modules listed in final stage (/etc/cloud/cloud.cfg) will execute.

Additional Info

Most cloud-init Python modules have a default run frequency: per_always (run on every boot), per_instance (run only if it has never run with this specific instance-id), per_boot (run only once, regardless of the instance-id).
Cloud-init keeps track of whether particular modules have been run by using semaphores. These get placed into the /var/lib/cloud/sem and /var/lib/cloud/sem/instances/<instance-id>/sem directories.
One can specify which specific modules to run at specific stages and override default module frequency:

#cloud-config
cloud_init_modules:
  - seed_random
  - [bootcmd, once]
  - [write-files, instance]
  - growpart
  - resizefs
  - set_hostname
  - update_hostname
  - rsyslog
  - users-groups
  - ssh

cloud_config_modules:
  - disable-ec2-metadata
  - runcmd
  - byobu

cloud_final_modules:
  - scripts-per-boot
  - scripts-per-instance
  - [scripts-user, always]

Overview of OpenStack Metadata Types

Matt Dorn — Mon, 02 May 2016 15:46:00 -0500

As a follow up to my recent workshop at the Austin OpenStack Summit, here is some of the information from this session:

When it comes to instance personalization on OpenStack, there are currently four primary types of metadata (these often show up on other cloud platforms as well): Meta-Data, User-Data, Vendor-Data, and Network-Data (https://github.com/openstack/nova/blob/7b032f5339d39ae3f57832c0285b2b818b1ec918/nova/api/metadata/base.py)

Meta-Data

Meta-Data is typically an instance-id, account-id, random seed, hostname, etc. It is found in the meta_data.json file residing on ConfigDrive or the Neutron Metadata Service via 169.254.169.254/openstack/latest/meta_data.json.

{
"admin_pass": "Amgkg8aehSeZ",
"availability_zone": "nova",
"hostname": "hi.novalocal",
"launch_index": 0,
"name": "hi",
"project_id": "b884b287309140b495821eb8f0f9e30d",
"random_seed": "FZHvd6YUHwDZUk4KYkVZkYrWHt61L1gGdbeoeefseeUUhmXvqCj9RfzKt9/RdD0DY2Fn48aAOVovSh6I6VtsRScV6ASjC+mpweJX1Jjmh73cjQnEfuN5YwUNknMYHZWDhWcKEjgRYpgM1BnaWYCt/wDVFC/Y0esLODy4156V94Sm8VhotDUawkoXT7BlCeLD+3bpAV1iprJn26CcM6Ob0ODKT3/qRVaSqxGD7qYi69RAelsneDOVn41xDJiQeIUqBCRzzahgDUoJ8HjFNuawFAqfez0Z3TqVsWuNHP03FhmNOG9Tqw7GevsMhLmzYY65ZHoCaJb/Ew64mDt57XyU+SZOeJDV9El9ENRQtO9vBmXttMpX5MIlsNsy/eeDTmfMZWxKsubmenMmtT7AUEO+PdPhTa/UMfnxCUX+ehFjUaPfTxTJJ8zBPyPoSMFpE5yA04z0G2U20Yt1Nisb4tt2UBKHJeDERjf1Ta7lE0Uc5HFBjg9U2LxZy1DKG18py7LajbsbCHfMyJtMM2RrhR/910VarUgscYhRo++EQzEAZfBwf+e3UFbmOCFrMP8ndq/jyyhGeGuPa7Rusd42Y70JZxnZTRfPQN9w2kSPxCu0l7RJ1+Ry/+rp77NDlhOHXZMf6a7CG8MbEbDslwtMMrFpCjcgTcY1ZETVvFFaqSg6uNM=",
"uuid": "f8b406c4-7da1-4d20-8c44-a93dc42308e2"
}

For security reasons, the admin_pass value is not typically used during most OpenStack deployments. One can create a customized cloud-init python module to fetch this admin_pass value and set it as root password. One can also choose to have libvirt mount the image and inject the password prior to boot.

random_seed is taken from the compute host's /dev/urandom.
launch_index is the number assigned to an instance when booting multiple servers via one api call or cli command, i.e.

nova boot --flavor 1 --image ubuntu --max-count 3 myinstance

myinstance-1 = launch_index: 0
myinstance-2 = launch_index: 1
myinstance-3 = launch_index: 2

User-Data

The user passes user-data at boot via the --user-data flag. It is found in the user_data file residing on ConfigDrive or the Neutron Metadata Service via 169.254.169.254/openstack/latest/user_data. User-data is typically cloud-config syntax. cloud-config is interepreted by cloud-init and provides an abstraction that is distro agnostic.

#cloud-config

package_update: true
package_upgrade: true

write_files:
- path: /etc/hosts
  permissions: '0644'
  content: |
    192.168.1.4 server2
    192.168.1.5 server3
    192.168.1.6 server4
    192.168.1.7 server5

packages:
  - vim
  - git

Each one of these directives (package_update/upgrade, write_files, packages, etc.) is directly related to a python module found in the cloud-init application (http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/cloudinit/config/). Cloud-init is typically found on any cloud image you boot, whether on Amazon, Rackspace, Google Cloud Platform, Azure, Digital Ocean, etc.

Cloud-init can also run a variety of other scripts (as long as the intereprter exists on the image). Examples include #!/bin/sh, #!/usr/bin/python, #!/usr/bin/perl, #!/usr/bin/awk.

User-Data is typically limited to 16K so you can also to the following to fetch the script and overcome the limitation:

#include
http://yoursite.com/yourscript.txt

Vendor-Data

Vendor-Data is a way for a vendor to provide site specific information. This can be information on local mirrors, a local proxy, or a one-time registration code. It is found in the vendor_data.json file residing on ConfigDrive or the Neutron Metadata Service via 169.254.169.254/openstack/latest/vendor_data.json. By default, Cloud-Init executes the Vendor-Data on every boot.

When running OpenStack, one can setup Vendor Data by adding the following lines to nova-api's /etc/nova/nova.conf:

[DEFAULT]
vendordata_driver=nova.api.metadata.vendordata_json.JsonFileVendorData
vendordata_jsonfile_path=/etc/nova/vendordata.json

The vendordata.json file can be any cloud-init readable data in json format. Example:

{
"cloud-init": "#cloud-config\nfinal_message: This is Vendor Data for the Austin OpenStack Summit Workshop by Rackspace Training"
}

Network-Data

Network-Data is network information found on ConfigDrive. It resides in the network_data.json file. Nova retrieves this info from Neutron. It includes fixed ip addresses, MAC addresses, port-id's, network-id's, subnet-id's, DNS name-servers, etc.

Consider a scenario where DHCP is disabled in the cloud. One could create a basic script or cloud-init module to assign static IP address found in this data to eth0. Or perhaps DHCP is enabled and ones to statically assign eth1 and eth2.

{
"links": [
        {
        "ethernet_mac_address": "fa:16:3e:9c:bf:3d",
        "id": "tapcd9f6d46-4a",
        "mtu": null,
        "type": "bridge",
        "vif_id": "cd9f6d46-4a3a-43ab-a466-994af9db96fc"
        }
],
"networks": [
        {
        "id": "network0",
        "link": "tapcd9f6d46-4a",
        "network_id": "99e88329-f20d-4741-9593-25bf07847b16",
        "type": "ipv4_dhcp"
        }
],
"services": [
        {
        "address": "8.8.8.8",
        "type": "dns"
        }
]
}

An Exploration of the OpenStack Metadata Service

Matt Dorn — Sun, 01 May 2016 12:00:00 -0500

Here's a video of my workshop on the OpenStack Metadata Service at the Austin OpenStack Summit.

Although it was a live hands-on workshop, the first 20-minutes provide the historical context for all current public/private clouds' metadata services.

Correction: I mention that the OpenStack config-drive is turned on by default in Nova. This is not true. One must ensure the following flag is set in /etc/nova/nova.conf on the nova-api server:

force_config_drive = True

If config-drive is set to false or not set at all, one can force it by using the --config-drive flag on boot:

nova boot --config-drive true --image my-image-name --flavor my-flavor myinstance

I will be posting all my cloud-init content from this workshop in some upcoming posts.

VirtualBox for the OpenStack Liberty Install Guide

Matt Dorn — Tue, 17 Nov 2015 16:20:00 -0600

The official OpenStack Liberty Install Guides are a great way to get hands-on experience with installing and configuring OpenStack by hand. This post describes how to install and configure a local VirtualBox environment for performing the step-by-step deployment instructions found in the guide.

Minimum Hardware Requirements

2 GHz or faster 64-bit (x64) processor with Intel VTx or AMD-V support
5 GB RAM or higher
20 GB or more available hard disk space

Download and Install VirtualBox

Navigate to VirtualBox Downloads.
Download and install the latest version of VirtualBox for your host operating system.

Configure VirtualBox Networking

Launch the VirtualBox application.
From the VirtualBox menu bar, select Preferences.
From the VirtualBox Preferences menu, select the Network icon.
Ensure the Nat Networks tab is highlighted and select the + icon:
This will automatically create a Nat Network named NatNetwork.
Select the tool icon to modify NatNetwork.
Rename the network to Management and change the Network CIDR to 10.0.0.0/24. Verify the Supports DHCP box is checked:

Note

The gateway for the Management network is 10.0.0.1.
Select Port Forwarding.
Select the + icon on the right to create a new rule. Name: SSH, Protocol: TCP, Host IP: 127.0.0.1, Host Port: 2022, Guest IP: 10.0.0.11, and Guest Port: 22. Select OK to save your changes:
Select the + icon to create another Nat Network. This will automatically create a Nat Network named NatNetwork1.
Select the tool icon to modify NatNetwork1.
Rename the network to Public and change the Network CIDR to 203.0.113.0/24. Uncheck the Supports DHCP box. Select OK to save your changes:

Note

The gateway for the Public network is 203.0.113.1.
You should now have two VirtualBox Nat Networks: Management and Public. Select OK to save your changes:

Download Operating System ISO

Navigate to the downloads pages for the operating system to be installed.
Download ISO.

Create the Controller Virtual Machine

From the VirtualBox toolbar, select the New button.
Type Controller in the Name section. The type is Linux. Select the version of Operating System you are installing:
Set the memory allocation to 2048 MB:
Select VDI (VirtualBox Disk Image).
Select Dynamically Allocated and size 5GB:

Modify Controller Virtual Machine Settings

Highlight the controller virtual machine and click the Settings button in the VirtualBox toolbar.
Click the System icon and click on the Acceleration tab. Ensure Paravirtualzation Inteface is Default and Enable VT-x/AMD-V and Enable Nested Paging are checked:

Note

If the Acceleration tab is grayed out, enable Virtualization Technology (Intel VTx or AMD-V) in the host machine's BIOS.
Select the Storage icon. Select the cd-rom icon titled Empty. In the Attributes section on the right, select the cd-rom icon and select Choose Virtual Optical Disk File from the drop down. Select your operating system iso:
Select the Network icon and click on the Adapter 1 tab. Select Nat Network from the Attached to section. In the Name section, select Management:
Select the Adapter 2 tab and ensure Enable Network Adapter is checked. Select Nat Network from the Attached to section. In the Name section, select Public. Select OK to save your settings:

Start and Install Controller Virtual Machine

Highlight the controller virtual machine and select the Start button. The virtual machine will begin to boot.
Follow the installation instructions for the operating system. Ensure SSH is installed and enabled.
Refer to Host Networking section of the Install Guide for instructions on configuring the Management and Public interfaces for the controller node.

Create the Compute Virtual Machine

From the VirtualBox toolbar, click the New button.
Type Compute in the Name section. The type is Linux. Select the version of Operating System you are installing:
Set the memory allocation to 2048 MB:
Select VDI (VirtualBox Disk Image).
Select Dynamically Allocated and size 10GB:

Modify Compute Virtual Machine Settings

Highlight the controller virtual machine and click the Settings button in the VirtualBox toolbar.
Click the System icon and click on the Acceleration tab. Ensure Paravirtualzation Inteface is Default and Enable VT-x/AMD-V and Enable Nested Paging are checked:

Note

If the Acceleration tab is grayed out, enable Virtualization Technology (Intel VTx or AMD-V) in the host machine's BIOS.
Select the Storage icon. Select the cd-rom icon titled Empty. In the Attributes section on the right, select the cd-rom icon and select Choose Virtual Optical Disk File from the drop down. Select your operating system iso:
Select the Network icon and click on the Adapter 1 tab. Select Nat Network from the Attached to section. In the Name section, select Management:
Select the Adapter 2 tab and ensure Enable Network Adapter is checked. Select Nat Network from the Attached to section. In the Name section, select Public. Select OK to save your settings:

Start and Install Compute Virtual Machine

Highlight the compute virtual machine and select the start button. The virtual machine will begin to boot.
Follow the installation instructions for your specific operating system.
Refer to Host Networking section of the Install Guide for instructions on configuring the Management and Public interfaces for the controller node.

Verify Connectivity

Verify SSH connectivity to the controller and compute nodes.

Note

This section assumes the network interfaces and name resolution on controller and compute have been configured per the steps outlined in Host Networking section of the Install Guide.

From terminal software on the host machine, SSH into the controller node:
```
$ ssh root@127.0.0.1 -p 2022
```
From the controller node, verify ssh connectivity to the compute node:
```
$ ssh root@compute1
```

RabbitMQ 3.4.3 Guest User Access with OpenStack Kilo

Matt Dorn — Fri, 08 May 2015 19:33:00 -0500

If one attempts to install OpenStack Kilo via the Ubuntu Cloud Archive, you will get Rabbit 3.4.3.

All RabbitMQ versions after 3.3.0 prohibit the guest user from connecting remotely by default so you must create a config file to allow this.

Don't use the guest user in a production OpenStack environment.

If you are setting up OpenStack Kilo for dev/testing purposes do the following:

Install RabbitMQ-Server:

apt-get install rabbitmq-server

Create the config file to allow the guest user to connect from anywhere:

cat > /etc/rabbitmq/rabbitmq.config <<EOF
[{rabbit, [{loopback_users, []}]}].
EOF

Restart RabbitMQ. Rabbit listens on 5672 by default.

service rabbitmq-server restart

Optional:

Enable the RabbitMQ API for management and monitoring:

rabbitmq-plugins enable rabbitmq_management

Restart RabbitMQ. The API will listen on 15672 by default.

service rabbitmq-server restart

Verify that you can authenticate against the RabbitMQ API with u:guest p:guest

curl -i -u guest:guest yourip:15672/api/whoami

Open up a web browser and connect to the Rabbit Management portal at http://yourip:15672 with u:guest p:guest.

Juno Neutron-Server Not Starting

Matt Dorn — Mon, 17 Nov 2014 15:11:00 -0600

The neutron-server service behavior in Juno has changed a bit since Icehouse.

Prior to Juno, when one went to start up neutron-server for the first time, an initial db sync was performed which generates the database schema.

One who attempts to intall Neutron on Juno without running a neutron-db-manage will no be able to start neutron-server and may see the following error or similar in /var/log/neutron/server.log:

ProgrammingError: (ProgrammingError) (1146, "Table 'neutron.ml2_gre_allocations' doesn't exist") 'SELECT ml2_gre_allocations.gre_id AS ml2_gre_allocations_gre_id, ml2_gre_allocations.allocated AS ml2_gre_allocations_allocated \nFROM ml2_gre_allocations' ()

To fix the issue, run the following prior to starting neutron-server:

sudo neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade juno

Installing VMware Tools on Centos7

Matt Dorn — Sun, 16 Nov 2014 15:11:00 -0600

If one attempts to install VMware Tools (VMwareTools-9.6.2-1688356) on Centos7 running the 3.10 kernel (i.e. 3.10.0-123.el7.x86_64), you will most likely see the following error:

In file included from /tmp/modconfig-5vxwsx/vmhgfs-only/inode.c:36:0:
/tmp/modconfig-5vxwsx/vmhgfs-only/inode.c: In function ‘HgfsPermission':
/tmp/modconfig-5vxwsx/vmhgfs-only/./shared/compat_dcache.h:57:38: error: ‘struct dentry’ has no member named ‘d_count’
#define compat_d_count(dentry) dentry->d_count
/tmp/modconfig-5vxwsx/vmhgfs-only/inode.c:1904:23: note: in expansion of macro ‘compat_d_count

Although most of the features will get installed, one will not have the ability to use VMware Shared Folders for file sharing between guest and host.

The issue is due to a VMware Tools bug which we can repair via a simple script. This script will replace a reference to kernel 3.11 with kernel 3.10 and then run the installer with default options for non-interactive installation.

Verify that you have the following packages installed before running:

yum -y install kernel-devel gcc dracut make perl

#!/bin/bash

mkdir /tmp/vmfusion
mkdir /tmp/vmfusion-archive
mount /dev/sr0 /tmp/vmfusion
tar xzfv /tmp/vmfusion/VMwareTools-*.tar.gz -C /tmp/vmfusion-archive
tar xfv /tmp/vmfusion-archive/vmware-tools-distrib/lib/modules/source/vmhgfs.tar -C /tmp/vmfusion-archive/vmware-tools-distrib/lib/modules/source/
sed -i -e '/KERNEL_VERSION/{s/3, 11, 0/3, 10, 0/}' /tmp/vmfusion-archive/vmware-tools-distrib/lib/modules/source/vmhgfs-only/shared/compat_dcache.h
rm -rf /tmp/vmfusion-archive/vmware-tools-distrib/lib/modules/source/vmhgfs.tar
tar cfv /tmp/vmfusion-archive/vmware-tools-distrib/lib/modules/source/vmhgfs.tar -C /tmp/vmfusion-archive/vmware-tools-distrib/lib/modules/source/ vmhgfs-only
rm -rf /tmp/vmfusion-archive/vmware-tools-distrib/lib/modules/source/vmhgfs-only/
/tmp/vmfusion-archive/vmware-tools-distrib/vmware-install.pl --default
umount /tmp/vmfusion
rm -rf  /tmp/vmfusion
rm -rf  /tmp/vmfusion-archive

If you exerienced this issue while attempting to build a Vagrant box with Packer, here is a shell provisioner script that I've tested and seems to be working fine: vmtools.sh

Nic Bonding in Ubuntu with VirtualBox

Matt Dorn — Mon, 06 Oct 2014 11:50:00 -0500

I recently needed to setup nic bonding in Virtualbox to demo software intended to be deployed in a nic bonded environment.

When doing this in VirtualBox, one needs to activiate the bond-fail-over-mac parameter to prevent issues with host-to-vm communication.

Here's the VirtualbBox network config:

Name:            node1
Guest OS:        Ubuntu (64 bit)
UUID:            cc550cfd-60b9-427c-b09d-0a2e0dad932b
NIC 1:           MAC: 08002760454A, Attachment: Host-only Interface 'vboxnet0'
NIC 2:           MAC: 080027A206C5, Attachment: Host-only Interface 'vboxnet0'

Here is the Ubuntu config in /etc/network/interfaces. Notice the bond-fail-over-mac flag. This will ensure that only one of the nic slaves is assigned the bond0 mac address at a given time, thus resolving any host-to-vm routing issues.

# The loopback network interface
auto lo
iface lo inet loopback

# Physical interface 0
auto eth0
iface eth0 inet manual
bond-master bond0
bond-primary eth0

# Physical interface 1
auto eth1
iface eth1 inet manual
bond-master bond0

# Bond interface 0: eth0 (vboxnet0), eth2 (vboxnet0)
auto bond0
iface bond0 inet static
bond-slaves none
bond-mode active-backup
bond-miimon 100
bond-downdelay 200
bond-updelay 200
bond-fail-over-mac 1
address 192.168.56.56
netmask 255.255.255.0

Configure Keystone to Utilize Apache

Matt Dorn — Fri, 22 Aug 2014 12:11:00 -0500

Keystone and many current OpenStack API components run in an Eventlet based http server. Eventlet is designed to perform well in networked environments and handles everything in a single thread.

The developers responsible for the Keystone project have recently recommended using Apache (with the mod_wsgi module) as a front-end rather than the traditional “Keystone” Eventlet-based process.

By using Apache as the front-end for Keystone, one gains better performance due to Apache’s ability to do multithreading. One can also take advantage of the variety of http server modules currently available for Apache. One popular module, Shibboleth, provides the ability to use one set of credentials to authenticate against multiple OpenStack clouds (more info here).

Here is a straight forward guide on how to setup Keystone to utilize Apache in your existing OpenStack deployment.

Stop Keystone Service (It is not necessary to run this service with this config)

sudo service keystone stop

Install Apache

sudo apt-get install -y apache2

Install Python WSGI module for Apache

sudo apt-get install -y libapache2-mod-wsgi

Make cgi-bin directory for Keystone

sudo mkdir -p /var/www/cgi-bin/keystone/

Create Python script for Apache (admin and main)

( cat | sudo tee /var/www/cgi-bin/keystone/admin /var/www/cgi-bin/keystone/main ) <<EOF
import logging
import os

from paste import deploy

from keystone.openstack.common import gettextutils
# NOTE(dstanek): gettextutils.enable_lazy() must be called before
# gettextutils._() is called to ensure it has the desired lazy #lookup behavior. This includes cases, like keystone.exceptions, #where gettextutils._() is called at import time.
gettextutils.enable_lazy()

from keystone.common import dependency
from keystone.common import environment
from keystone.common import sql
from keystone import config
from keystone.openstack.common import log
from keystone import service


CONF = config.CONF

config.configure()
sql.initialize()
config.set_default_for_default_log_levels()

CONF(project='keystone')
config.setup_logging()

environment.use_stdlib()
name = os.path.basename(__file__)

if CONF.debug:
CONF.log_opt_values(log.getLogger(CONF.prog), logging.DEBUG)


drivers = service.load_backends()

# NOTE(ldbragst): 'application' is required in this context by WSGI spec.
# The following is a reference to Python Paste Deploy documentation
# http://pythonpaste.org/deploy/
application = deploy.loadapp('config:%s' % config.find_paste_config(),
                     name=name)

dependency.resolve_future_dependencies()
EOF

Configure Apache to listen on ports 35357(admin) and 5000(main)

( cat | sudo tee /etc/apache2/ports.conf ) <<EOF
Listen 35357
Listen 5000
EOF

Configure Keystone Virtual Hosts

( cat | sudo tee /etc/apache2/sites-available/keystone-httpd.conf ) <<EOF
WSGIDaemonProcess keystone user=keystone group=nogroup processes=3 threads=10

<VirtualHost *:5000>
    LogLevel  info
    ErrorLog  /var/log/keystone/keystone-apache-error.log
    CustomLog /var/log/keystone/ssl_access.log combined
    Options +FollowSymLinks

#SSLEngine on
#SSLCertificateFile /etc/ssl/certs/mycert.pem
#SSLCertificateKeyFile /etc/ssl/private/mycert.key
#SSLVerifyClient optional
#SSLVerifyDepth 10
#SSLProtocol all -SSLv2
#SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
#SSLOptions +StdEnvVars +ExportCertData

    WSGIScriptAlias /  /var/www/cgi-bin/keystone/main
    WSGIProcessGroup keystone
</VirtualHost>

<VirtualHost *:35357>
    LogLevel  info
    ErrorLog  /var/log/keystone/keystone-apache-error.log
    CustomLog /var/log/keystone/ssl_access.log combined
    Options +FollowSymLinks

#SSLEngine on
#SSLCertificateFile /etc/ssl/certs/mycert.pem
#SSLCertificateKeyFile /etc/ssl/private/mycert.key
#SSLVerifyClient optional
#SSLVerifyDepth 10
#SSLProtocol all -SSLv2
#SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
#SSLOptions +StdEnvVars +ExportCertData

    WSGIScriptAlias / /var/www/cgi-bin/keystone/admin
    WSGIProcessGroup keystone
</VirtualHost>
EOF

Enable Keystone site

sudo a2ensite keystone-httpd

Reload Apache

sudo service apache2 reload

Adjust File Injection Upon Boot Quota

Matt Dorn — Thu, 29 May 2014 16:03:00 -0500

One way to customize the personality of an instance is by injecting data into it upon boot

This would be handy for updating /etc/passwd or /etc/group with customized pre-built files.

Here is an example:

nova boot --image cirros-qcow2 --flavor m1.tiny  --file /etc/passwd=/your/local/path/passwd MyInstance

By default, the Icehouse release of OpenStack can support up 5 files:

nova boot --image cirros-qcow2 --flavor m1.tiny  --file /home/cirros/myfirstfile=myfirstfile --file /home/cirros/mysecondfile=mysecondfile --file /home/cirros/mythirdfile=mythirdfile --file /home/cirros/myfourthfile=myfourthfile --file /home/cirros/myfifthfile=myfifthfile MyInstance

If you would like to inject more than 5 files, update your /etc/nova/nova.conf with the following flag:

quota_injected_files=50

Restart nova-compute

sudo service nova-compute restart

50 appears to the max amount of files it will allow you to pass.

Additional info about file injection:

Max size per file is 255 bytes.
One can only inject text files. Binary or zip files won't work.
During file injection, any existing files that match specified files are renamed to include .bak extension appended with a time stamp.
Once written, the files have 0440 permissions.

Glance Direct URL Metadata

Matt Dorn — Fri, 07 Mar 2014 19:52:00 -0600

OpenStack Havana comes with Glance's v1 and v2 API.

python-glanceclient fully supports v1 while support for v2 is in progress.

v2 adds some cool stuff including:

The ability to share images with specific users or projects
Image tagging
Schemas: Gets a JSON-schema document that represents an image or image-entity

One particular useful feature is direct-url metadata which allows one to see where a particular image resides.

Here's how to enable it:

Modify /etc/glance/glance-api.conf

show_image_direct_url = True

Restart Glance Services

service glance-api restart
service glance-registry restart

Utilize v2 API to Display Direct-Url

glance --os-image-api-version 2 image-show b9f44c5c-fc27-4891-b246-2df086f32ed7

+------------------+--------------------------------------------------------------------+
| Property         | Value                                                              |
+------------------+--------------------------------------------------------------------+
| checksum         | 4296b4c64655746fd44b7239fb66cf07                                   |
| container_format | bare                                                               |
| created_at       | 2014-03-07T23:30:00Z                                               |
| direct_url       | file:///var/lib/glance/images/b9f44c5c-fc27-4891-b246-2df086f32ed7 |
| disk_format      | qcow2                                                              |
| id               | b9f44c5c-fc27-4891-b246-2df086f32ed7                               |
| min_disk         | 0                                                                  |
| min_ram          | 0                                                                  |
| name             | cirros-qcow2                                                       |
| protected        | False                                                              |
| size             | 13139456                                                           |
| status           | active                                                             |
| tags             | []                                                                 |
| updated_at       | 2014-03-07T23:30:00Z                                               |
| visibility       | public                                                             |
+------------------+--------------------------------------------------------------------+

Intro to Keystone v3 API

Matt Dorn — Mon, 20 Jan 2014 22:04:00 -0600

The Keystone v3 API is included with OpenStack Grizzly & Havana but the Python-KeystoneClient does not expose v3 functionality at this time. Any interaction with the v3 API needs to be done directly via curl or tool of your choice. The term tenant is now officially replaced with project.

The Keystone v3 API introduces two significant Keystone features/concepts:

Domains
Groups

Let's first take a look at the Keystone v2 model.

Example Org Structure Utilizing Keystone v2 API

A user can reside in multiple departments yet have a different role in that department.
SandraD is a Sysadmin in Aerospace, but not in Biology. In the CompSci project, she has the Support role, but not the Sysadmin role.

Example of v3 API "Domains"

v3 now introduces true multi-tenancy with the use of domains. As we can see the domain acts as a high level container for projects.

With domains, a cloud customer can be the owner of the domain. They can then create additional users, groups, and roles to be used within their specified domain.

Example of v3 API "Groups"

A group is simply a container representing a collection of users. Rather than assign a role directly to user/project, a domain owner can assign a role to a group, and then add users to that group.

A role can be assigned to a domain on a group or a project on a group.

In this example:

JohnB will authenticate with Keystone, obtain the Sysadmin role and belong to Biology, Aerospace, and Compsci projects.
LisaD will authenticate with Keystone, obtain the Engineer role and only belong to the Compsci project.

Other Info:

Groups are optional.
The domain name and role name is globally unique across all domains.
The username, project, and group name are only unique to the owning domain.

New v3 Commands

Create a group
Delete a group
Update a group (change its name or description)
Add a user to a group
Remove a user from a group
List group members
List groups for a user
Assign a role on a tenant to a group
Assign a role on a domain to a group
Query role assignments to groups

Interacting with Keystone v3 API

Adam Young's blog has some examples of interacting with v3: http://adam.younglogic.com/2013/09/keystone-v3-api-examples/

Resize a Running Instance on a Single Compute Node

Matt Dorn — Wed, 18 Dec 2013 20:22:00 -0600

One can resize a running instance in OpenStack by running the following nova command:

nova resize --poll <server> <flavor>

The default behavior of this command is to invoke the nova scheduler to determine the best compute node for the resized instance. If there is only one compute node in the environment, the resize will fail.

We need to tell nova to allow resizing on the same host by modifying /etc/nova/nova.conf .

Modify /etc/nova/nova.conf

allow_resize_to_same_host=True
scheduler_default_filters=AllHostsFilter

By changing the default nova scheduler filter to AllHostsFilter , all compute hosts will be available and unfiltered. It is not a good idea to keep this setting in a multi-compute environment.

Restart Services

After making the changes to nova.conf, restart scheduler and compute services:

service nova-scheduler restart
service nova-compute restart

Resize Confirm

After resizing the instance, the status will change to VERIFY_RESIZE . One will see two copies of the instance in /var/lib/nova/instances : the original and the resized. Run the following command to verify the resize and delete the original instance:

nova resize-confirm <server>

Quickly Delete a Large Cloud Files Container

Matt Dorn — Tue, 29 Oct 2013 02:44:00 -0500

As you probably know, one cannot delete a Cloud Files container without deleting all the objects inside it first. This can take quite a bit of time if you have many objects in your container.

I had to delete about 30GB of data in a container and it was taking forever. I discovered a script by Fernando Florez which utilizes the Python Gevent library and Pyrax SDK to do a speedy container deletion. The Gevent library contains Greenlet which does micro-threading. Executing this script sends deletion requests to Cloud Files concurrently, drastically reducing total deletion time.

This method assumes you are utilizing a Python virtual environment and that you have activated it.

Install Pyrax

pip install pyrax

Install Libevent

curl -L -O https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz
tar xzf libevent-2.0.21-stable.tar.gz
cd libevent-2.0.21-stable
./configure --prefix="$VIRTUAL_ENV"
make && make install
cd $VIRTUAL_ENV/..

Install Greenlet and Gevent

pip install greenlet
pip install gevent --global-option "-L$VIRTUAL_ENV/lib"  --global-option "-I$VIRTUAL_ENV/include"

Test Gevent

Enter a Python interactive shell and try to import Gevent. If you do not get any errors you should be good.

python
import gevent

Delete All Objects in Container

Be sure to specify your region, i.e. DFW, ORD, etc.

from gevent import monkey
from gevent.pool import Pool
from gevent import Timeout
monkey.patch_all()
import pyrax

if __name__ == '__main__':
pool = Pool(100)
pyrax.set_setting('identity_type', 'rackspace')
pyrax.set_setting('verify_ssl', False)
pyrax.set_setting("region", 'region')
pyrax.set_credentials('username', 'api_key')

cf = pyrax.cloudfiles
container = cf.get_container('container_name')
objects = container.get_objects(full_listing=True)
def delete_object(obj):
# added timeout of 5 seconds just in case
with Timeout(5, False):
          try:
              obj.delete()
          except:
              pass
  for obj in objects:
      pool.spawn(delete_object, obj)
  pool.join()

How Keystone Signs, Encodes, Decodes, & Verifies

Matt Dorn — Sun, 13 Oct 2013 10:20:00 -0500

There seemed to be a great deal of confusion about whether Keystone PKI tokens were encrypted or signed.

I spent a few days digging into this.

Initial Setup: Keystone Creates a Self-Signed Certificate and Becomes the Root CA

Utilizing OpenSSL, Keystone sets itself up as the root CA by generating a self-signed cert:

openssl genrsa -out /etc/keystone/ssl/certs/cakey.pem 1024 -config /etc/keystone/ssl/certs/openssl.conf
openssl req -new -x509 -extensions v3_ca -passin pass:None -key /etc/keystone/ssl/certs/cakey.pem -out /etc/keystone/ssl/certs/ca.pem -days 3650 -config /etc/keystone/ssl/certs/openssl.conf -subj /C=US/ST=Unset/L=Unset/O=Unset/CN=www.example.com

Generate Signing Certificate

Now that Keystone has established itself as the root CA, it is ready to generate legitimate certs. We will generate a signing certificate. This certificate is going to be used to sign our token info:

openssl req -out /etc/keystone/ssl/certs/req.pem -config /etc/keystone/ssl/certs/openssl.conf -subj /C=US/ST=Unset/L=Unset/O=Unset/CN=www.example.com
openssl ca -batch -out /etc/keystone/ssl/certs/signing_cert.pem -config /etc/keystone/ssl/certs/openssl.conf -infiles /etc/keystone/ssl/certs/req.pem

If we did not want to use Keystone as the root CA, we could generate a request here and send it off to an established CA like Verisign. Verisign would then send back our signing cert.

Initial Authentication

When a user initially authenticates with Keystone, it will authenticate the user and grab data about the user from the Keystone DB:

{
 "access": {
     "metadata": {
         "is_admin": 0,
         "roles": [
             "9fe2ff9ee4384b1894a90878d3e92bab",
             "9ccd9139439e41ceac20a5e33e81555b"
         ]
     },
     "serviceCatalog": [
         {
             "endpoints": [
                 {
                     "adminURL": "http://10.0.10.10:8776/v1/5dd0ede8bcc44f26ae8983f03249b4de",
                     "id": "4be86b9f26564ffc9b40fa1f68eb1603",
                     "internalURL": "http://10.0.10.10:8776/v1/5dd0ede8bcc44f26ae8983f03249b4de",
                     "publicURL": "http://10.0.10.10:8776/v1/5dd0ede8bcc44f26ae8983f03249b4de",
                     "region": "RegionOne"
                 }
             ],
             "endpoints_links": [],
             "name": "volume",
             "type": "volume"
         },
         {
             "endpoints": [
                 {
                     "adminURL": "http://10.0.10.10:9292",
                     "id": "5655ca5092b14a63a5a6a1882426bce6",
                     "internalURL": "http://10.0.10.10:9292",
                     "publicURL": "http://10.0.10.10:9292",
                     "region": "RegionOne"
                 }
             ],
             "endpoints_links": [],
             "name": "glance",
             "type": "image"
         },
         {
             "endpoints": [
                 {
                     "adminURL": "http://10.0.10.10:8774/v2/5dd0ede8bcc44f26ae8983f03249b4de",
                     "id": "5721a24df1c3492da5532ed71d13ebe0",
                     "internalURL": "http://10.0.10.10:8774/v2/5dd0ede8bcc44f26ae8983f03249b4de",
                     "publicURL": "http://10.0.10.10:8774/v2/5dd0ede8bcc44f26ae8983f03249b4de",
                     "region": "RegionOne"
                 }
             ],
             "endpoints_links": [],
             "name": "nova",
             "type": "compute"
         },
         {
             "endpoints": [
                 {
                     "adminURL": "http://10.0.10.10:9696",
                     "id": "5ab1bbb07fa644a0b95c1a1cad3061be",
                     "internalURL": "http://10.0.10.10:9696",
                     "publicURL": "http://10.0.10.10:9696",
                     "region": "RegionOne"
                 }
             ],
             "endpoints_links": [],
             "name": "quantum",
             "type": "network"
         },
         {
             "endpoints": [
                 {
                     "adminURL": "http://10.0.10.10:35357/v2.0",
                     "id": "6e89262f99b54900a066d1310c2e4c87",
                     "internalURL": "http://10.0.10.10:5000/v2.0",
                     "publicURL": "http://10.0.10.10:5000/v2.0",
                     "region": "RegionOne"
                 }
             ],
             "endpoints_links": [],
             "name": "keystone",
             "type": "identity"
         }
     ],
     "token": {
         "expires": "2013-10-12T20:02:49Z",
         "id": "placeholder",
         "issued_at": "2013-10-11T20:02:49.623814",
         "tenant": {
             "description": "Default Tenant",
             "enabled": true,
             "id": "5dd0ede8bcc44f26ae8983f03249b4de",
             "name": "demo"
         }
     },
     "user": {
         "id": "a366ed02b00f4cc6ac9af6f6d5990778",
         "name": "admin",
         "roles": [
             {
                 "name": "_member_"
             },
             {
                 "name": "admin"
             }
         ],
         "roles_links": [],
         "username": "admin"
     }
 }
}

Sign & Encode

Keystone then signs the data above with both the signing cert and associated signing private key:

openssl cms -sign -in mytext.json -signer /etc/keystone/ssl/certs/signing_cert.pem -inkey /etc/keystone/ssl/private/signing_key.pem -outform PEM -nosmimecap -nodetach -nocerts -noattr

The result is our signed info encoded, not encrypted, in PEM format:

-----BEGIN CMS-----
MIIJWAYJKoZIhvcNAQcCoIIJSTCCCUUCAQExCTAHBgUrDgMCGjCCCDEGCSqGSIb3
DQEHAaCCCCIEgggeeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAi
MjAxMy0xMC0xMVQyMDowMjo0OS42MjM4MTQiLCAiZXhwaXJlcyI6ICIyMDEzLTEw
LTEyVDIwOjAyOjQ5WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7
ImRlc2NyaXB0aW9uIjogIkRlZmF1bHQgVGVuYW50IiwgImVuYWJsZWQiOiB0cnVl
LCAiaWQiOiAiNWRkMGVkZThiY2M0NGYyNmFlODk4M2YwMzI0OWI0ZGUiLCAibmFt
ZSI6ICJkZW1vIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBb
eyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4xMC4xMDo4Nzc2L3YxLzVkZDBlZGU4
YmNjNDRmMjZhZTg5ODNmMDMyNDliNGRlIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUi
LCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMTAuMTA6ODc3Ni92MS81ZGQw
ZWRlOGJjYzQ0ZjI2YWU4OTgzZjAzMjQ5YjRkZSIsICJpZCI6ICI0YmU4NmI5ZjI2
NTY0ZmZjOWI0MGZhMWY2OGViMTYwMyIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEw
LjAuMTAuMTA6ODc3Ni92MS81ZGQwZWRlOGJjYzQ0ZjI2YWU4OTgzZjAzMjQ5YjRk
ZSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJ2b2x1bWUiLCAi
bmFtZSI6ICJ2b2x1bWUifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJo
dHRwOi8vMTAuMC4xMC4xMDo5MjkyIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAi
aW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjAuMTAuMTA6OTI5MiIsICJpZCI6ICI1
NjU1Y2E1MDkyYjE0YTYzYTVhNmExODgyNDI2YmNlNiIsICJwdWJsaWNVUkwiOiAi
aHR0cDovLzEwLjAuMTAuMTA6OTI5MiJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtd
LCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50
cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4wLjEwLjEwOjg3NzQvdjIvNWRk
MGVkZThiY2M0NGYyNmFlODk4M2YwMzI0OWI0ZGUiLCAicmVnaW9uIjogIlJlZ2lv
bk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMC4xMC4xMDo4Nzc0L3Yy
LzVkZDBlZGU4YmNjNDRmMjZhZTg5ODNmMDMyNDliNGRlIiwgImlkIjogIjU3MjFh
MjRkZjFjMzQ5MmRhNTUzMmVkNzFkMTNlYmUwIiwgInB1YmxpY1VSTCI6ICJodHRw
Oi8vMTAuMC4xMC4xMDo4Nzc0L3YyLzVkZDBlZGU4YmNjNDRmMjZhZTg5ODNmMDMy
NDliNGRlIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImNvbXB1
dGUiLCAibmFtZSI6ICJub3ZhIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwi
OiAiaHR0cDovLzEwLjAuMTAuMTA6OTY5NiIsICJyZWdpb24iOiAiUmVnaW9uT25l
IiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4wLjEwLjEwOjk2OTYiLCAiaWQi
OiAiNWFiMWJiYjA3ZmE2NDRhMGI5NWMxYTFjYWQzMDYxYmUiLCAicHVibGljVVJM
IjogImh0dHA6Ly8xMC4wLjEwLjEwOjk2OTYifV0sICJlbmRwb2ludHNfbGlua3Mi
OiBbXSwgInR5cGUiOiAibmV0d29yayIsICJuYW1lIjogInF1YW50dW0ifSwgeyJl
bmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMC4xMC4xMDozNTM1
Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAi
aHR0cDovLzEwLjAuMTAuMTA6NTAwMC92Mi4wIiwgImlkIjogIjZlODkyNjJmOTli
NTQ5MDBhMDY2ZDEzMTBjMmU0Yzg3IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAu
MC4xMC4xMDo1MDAwL3YyLjAifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5
cGUiOiAiaWRlbnRpdHkiLCAibmFtZSI6ICJrZXlzdG9uZSJ9XSwgInVzZXIiOiB7
InVzZXJuYW1lIjogImFkbWluIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJh
MzY2ZWQwMmIwMGY0Y2M2YWM5YWY2ZjZkNTk5MDc3OCIsICJyb2xlcyI6IFt7Im5h
bWUiOiAiX21lbWJlcl8ifSwgeyJuYW1lIjogImFkbWluIn1dLCAibmFtZSI6ICJh
ZG1pbiJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyI5
ZmUyZmY5ZWU0Mzg0YjE4OTRhOTA4NzhkM2U5MmJhYiIsICI5Y2NkOTEzOTQzOWU0
MWNlYWMyMGE1ZTMzZTgxNTU1YiJdfX19DQoxgf8wgfwCAQEwXDBXMQswCQYDVQQG
EwJVUzEOMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVV
bnNldDEYMBYGA1UEAxMPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqG
SIb3DQEBAQUABIGACOF/D6COhzYHNdpJmE+PCz7TxkgnVBuHjxU2z/pV1e8hwv1I
VO2ZTJ+TRIYZl8LxFCtupaDPeTOYEz1fn6YvjGOqSBkmw1OS3mmGov7HaH0BsYk3
vWnb0lXUNCvsuSzqQmoCzv3CbrsXDnp5bDEbzYXM78AHX03homw/PyAPA5M=
-----END CMS-----

View Signature

We can view the signature or "proof" that the message was stamped with Keystone's approval by decoding our token:

base64 -d mytoken.signed

The decoded token is our user metadata along with some scrambled text which represents the cert signaure:

?"{"access": {"token": {"issued_at": "2013-10-11T20:02:49.623814", "expires": "2013-10-12T20:02:49Z", "id": "placeholder"
,"tenant": {"description": "Default Tenant", "enabled": true, "id": "5dd0ede8bcc44f26ae8983f03249b4de", "name": "demo"}},
"serviceCatalog": [{"endpoints": [{"adminURL": "http://10.0.10.10:8776/v1/5dd0ede8bcc44f26ae8983f03249b4de", "region":
"RegionOne", "internalURL": "http://10.0.10.10:8776/v1/5dd0ede8bcc44f26ae8983f03249b4de", "id": "4be86b9f26564ffc9b40fa1f
68e
b1603", "publicURL": "http://10.0.10.10:8776/v1/5dd0ede8bcc44f26ae8983f03249b4de"}], "endpoints_links": [], "type":
"volume", "name": "volume"}, {"endpoints": [{"adminURL": "http://10.0.10.10:9292", "region": "RegionOne", "internalURL":
"http://10.0.10.10:9292", "id": "5655ca5092b14a63a5a6a1882426bce6", "publicURL": "http://10.0.10.10:9292"}], "endpoints_l
ink
s": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://10.0.10.10:8774/v2/5dd0ede8bcc44f26ae8983
f03
249b4de", "region": "RegionOne", "internalURL": "http://10.0.10.10:8774/v2/5dd0ede8bcc44f26ae8983f03249b4de", "id":
"5721a24df1c3492da5532ed71d13ebe0", "publicURL": "http://10.0.10.10:8774/v2/5dd0ede8bcc44f26ae8983f03249b4de"}],
"endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://10.0.10.10:9696", "region"
: "RegionOne", "internalUR   L": "http://10.0.10.10:9696", "id": "5ab1bbb07fa644a0b95c1a1cad3061be", "publicURL": "http
://10.0.10.10:9696"}], "endpoints_
links": [], "type": "network", "name": "quantum"}, {"endpoints": [{"adminURL": "http://10.0.10.10:35357/v2.0", "region":
"RegionOne", "internalURL": "http://10.0.10.10:5000/v2.0", "id": "6e89262f99b54900a066d1310c2e4c87", "publicURL": "http
://10
.0.10.10:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin",
"roles_links": [], "id": "a366ed02b00f4cc6ac9af6f6d5990778", "roles": [{"name": "_member_"}, {"name": "admin"}], "name":
"admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "9ccd9139439e41ceac20a5e33e81555b"]}}
}
1??0??0\0W1
0UUS10
Unset10
UUnset10
U
?????65?I?O??xample.com0+0
>??H'T?6??U??!??HT?L??D????+n???y3?=_??/?c?H&?S??i????h}??7?i??U?4+?,?Bj???n?zyl1ͅ??_M?l?? ?

Certs Pushed to Services

The signing certificate and Keystone root CA cert are passed down to the service endpoint along with the current list of revoked certificates.

Example:

/var/lib/glance
.
|-- image-cache
|   |-- incomplete
|   |-- invalid
|   `-- queue
|-- images
|   `-- b32c90b4-f30a-4dd0-8898-ee35d67838fd
`-- keystone-signing
   |-- cacert.pem
   |-- revoked.pem
   `-- signing_cert.pem

Token Verification

The token is verified by a service using a signing cert (public key) and Keystone root CA cert (The Keystone root CA cert is needed because it is not from an established CA and we are not likely to have a copy on our local machine):

openssl cms -verify -in cmsnew -certfile /var/lib/glance/keystone-signing/signing_cert.pem -CAfile /var/lib/glance/keystone-signing/cacert.pem -inform PEM -nosmimecap -nodetach -nocerts -noattr

This works by decoding and then verifiying the signature. The scrambled text is removed from our data and we now have our original user json metadata for OpenStack to consume.

Conclusion

If one does not have their SSL endpoints secure, nothing is being encrypted. It is simply being signed and encoded.

To prove this point, it's a cool exercise to just run base64 -d on any PKI token to show that the data inside is not safe! The PEM format is simply for transportation purposes. The "signing" is to prevent spoofing, i.e. modifying a token and adding the admin role.

Encrypting: Using a public key to write a message and someone else using their private key to read it. i.e. You sign into https://gmail.com: You view their certificate by clicking on the SSL lock encryption icon. You can see the public key they are providing to your browser to encrypt the data to them. They are using a hidden private key to decrypt.

Signing: You use your private key to write message's signature. They use your public key to check if it's really yours.

Check out this resource for clarification on the differences between encryption, encoding, and hashing:

http://danielmiessler.com/study/encoding_encryption_hashing/

Syncing QNAP to Rackspace Cloud Files

Matt Dorn — Wed, 09 Oct 2013 10:20:00 -0500

Users of QNAP NAS devices have been asking for an out-of-the-box solution to backup to Rackspace Cloud Files for a few years now. There is already a tool in the QNAP Web UI for backing up your shares to Amazon S3, Elephant Drive, or Symform.

The best solution I have so far is utilizing Pyrax, the Python SDK for Rackspace public cloud. The cf_pyrax script makes use of the sync_folder_to_container function found in Pyrax to sync a directory to a Cloud Files container. Even though the QNAP NAS is Linux-based, a lot of the usual methods for installing packages and scheduling jobs do not apply.

Here is what I did to configure my QNAP TS-419P II for a scheduled backup to Rackspace Cloud Files.

Install Python 2.7

Navigate to the QNAP web interface and install the Python v2.7 .qpkg from the App Center.

Verify the Installation

Verify that Python 2.7 was successfully installed:

python --version

Install PIP

Install PIP 1.2.1:

curl -o https//:pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz
tar xvfz pip-1.2.1.tar.gz
cd pip-1.2.1
python setup.py install

If you get "ImportError: No module named setuptools", install setuptools first:

wget https://pypi.python.org/packages/source/s/setuptools/setuptools-1.0.tar.gz --no-check-certificate
tar xf setuptools-1.0.tar.gz
cd setuptools-1.0
python setup.py build
python setup.py install

Install Pyrax

I had some issues with the NAS running out of room on tmpfs when installing Pyrax so provide a cache path with ample space:

pip install pyrax --download-cache /share/path/to/directory

Grab the cf_pyrax.py script from https://github.com/Linuturk/www.onitato.com/blob/master/cf_pyrax.py

Create a Credentials File

Create a credentials file with the following info:

[rackspace_cloud]
username = myusername
api_key = myapikey

Test Sync

Your syntax to sync to a specific container should be as follows:

python cf_pyrax.py TestContainer1 /share/MD0_DATA/testshare1 DFW /share/MD0_DATA/credentials.txt

Edit Crontab

After ensuring that the sync works properly, setup a cron job to backup your shares at a desired time.

QNAP overwrites crontab with etc/config/crontab after every system reboot/shutdown. You must modify this file if you want new cron job entries to stick:

echo "1 4 * * * python cf_pyrax.py TestContainer1 /share/MD0_DATA/testshare1 DFW /share/MD0_DATA/credentials.txt" >> /etc/config/crontab
echo "1 4 * * * python cf_pyrax.py TestContainer2 /share/MD0_DATA/testshare2 DFW /share/MD0_DATA/credentials.txt" >> /etc/config/crontab
crontab /etc/config/crontab