Thursday, January 31, 2019

Troubleshooting Kubernetes: "zombie pods"

I recently ran into a mysterious problem as I was developing and testing a lab exercise to teach about Kubernetes resiliency. I sort of caused the problem myself, because I had run through several scenarios with the example application, and I wanted to blow it all away and start over...so I just started deleting things. That, my friends, is a sure-fire way to break something. If you are dealing with Deployments and ReplicaSets, merely deleting a pod is just going to cause K8s to try and redeploy it.

I ended up with a handful of pods that were stuck in a state of "Terminating," and they would not die. For days. So, I asked around, tried researching the problem. A google search of "pods stuck in terminating" gave many hits, with many different possible causes and solutions. Some issues mentioned kubelet and a hostname mismatch - that was not it. 

I tried doing a drain, cordon, and shutdown of the node. When I started it back up, the pods were still there, still terminating. 

I tried deleting the Helm release, and helm command would not work. I got errors, which I googled, and that pointed to a problem with Tiller. I tried deleting the helm cache. I tried reinstalling Tiller, but Tiller would not come up. The Tiller pod was stuck in a state of "Pending." 

Some of the issues mentioned that they were trying to deploy pods on the master node, which is something you typically don't do, because the whole point of K8s is to let it schedule pods on the worker nodes. However, if you want a single-node environment, or something like that, you must remove a taint from the master node that prevent pods from being scheduled on it. That sounded a lot like my situation, because I was actually only running a master node...

...and that was the problem. I started with a cluster with two worker nodes, but they were offline. I had not restarted them since encountering the problem. After starting the workers, the status of all the hung pods resolved.

The moral of this story is: sometimes the answer is just too obvious to find through an internet search.  


Tuesday, June 12, 2018

IBM Microservices Specialization on Coursera

The IBM Microservices Specialization just launched on Coursera! This series of self-paced online courses takes you through microservices fundamentals, microservices application development, and deployment of microservices to IBM Cloud and IBM Cloud private. In this course, you get hands-on experience with:

  • Docker and containerized applications
  • Kubernetes - a container orchestrator
  • Helm - the Kubernetes package manager
  • Microclimate - an end-to-end development environment for microservices
  • Jenkins, and other tools used for DevOps
  • IBM Cloud (formerly known as Bluemix), and 
  • IBM Cloud Private - an application platform for developing and managing on-premises, containerized applications. 
You can audit this series at no charge, or enroll for a nominal fee and earn a completion certificate. 

Monday, February 12, 2018

Microservices and Kubernetes education on Coursera.org

I recently developed and launched a new course on Coursera.org -

IBM Cloud: Deploying Microservices with Kubernetes

About this course: In this course, you learn how to install the Kubernetes command-line interface (CLI), and create a Kubernetes cluster on which to run applications. Hands-on tutorials show you how to deploy microservices to a Kubernetes cluster. You also learn about securing and managing a Kubernetes cluster, and how to plan your Kubernetes cluster for deployment on IBM Cloud.

You can audit this course at no charge, or pay a nominal fee to receive the badge/certificate after completing it.

This course is part of a specialization on Microservices. If you complete all courses in the specialization, you get a special badge. Another course in the specialization, Microservices Fundamentals, also recently launched, and two more courses in the specialization are targeted for release later this year.

Coursera is a an education-focused technology company that offers online courses, and works with universities and other organizations to offer online courses, specializations, and degrees in a variety of subjects.

Thursday, July 20, 2017

Teaching mobile developers in Africa

This article is getting a replay on the IBM Mobile Business Insights blog:

http://mobilebusinessinsights.com/2017/07/sowing-the-seeds-for-tomorrows-mobile-developers-in-africa/

A bunch of my articles from that time period have been archived, and are no longer available online, but this one, I think, is worth revisiting. It was one of the most interesting and successful experiences that I had while teaching abroad.

Friday, December 11, 2015