Chapter 1 - Vagrantfile, Virtual Box, and Docker for Mac Installation
Docker for Mac の Bridgeモードを考察してみた. はじめに 【 大阪オフィス開設1周年勉強会 】第6回 本番で使うDocker勉強会 in 大阪 2017/06/09 #cmosaka | Developers.IO に参加した際、内輪で Docker の Bridge モードと Host モードの違いは何?. $ ln -s /var/run/docker/netns /var/run $ ls -l /var/run/netns lrwxrwxrwx 1 root root 21 Aug 31 01:21 /var/run/netns - /var/run/docker/netns. 通过ARP协议,可以找到目的IP地址172.17.0.3的MAC地址,也就是container B的网络空间中的eth0的网卡. #Create network Docker network create -d macvlan -options macnet14 Docker run -net=macnet14 Deep Dive. Manager nodes in swam maintain the network. Network Control Plane. Gossip based protocol Hosts contain gossiped networks; Clusters have a higher level scoop that contains host networks; Gossip can scale to any number of nodes (10,000. $ docker run -d -network testnet1 -ip 172.21.0.2 -name test alpine sleep 6000 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c888cb905dcf alpine 'sleep 6000' About a minute ago Up About a minute test $ docker exec -ti test hostname -i 172.21.0.2.
- Download Vagrant Up for Machttps://www.vagrantup.com/downloads.html
- Download Virtual Box for Machttps://www.virtualbox.org/wiki/Downloads
- Download Docker Toolbox for Machttps://www.docker.com/products/docker-toolbox
- Download Docker for Machttps://www.docker.com/products/docker#/mac
- Forum for Docker https://forums.docker.com/c/docker-for-mac
- Guide for Docker for Machttps://docs.docker.com/docker-for-mac/
- Examples for Docker for Machttps://docs.docker.com/docker-for-mac/examples/
- Setup Docker Compose and MySQLhttps://docs.docker.com/compose/rails/
Check Docker Client
Create CoreOS Docker host using Vagrant VM by:
- Note: Use of Vagrant seed for docker development is an alternative approach to using
docker-machine
. - Cloning coreos-vagrant Git Repo into host machine directory
docker-host
- Making the following changes to the Vagrantfile
- Expose Docker TCP connection port in
config.rb
of coreos-vagrant directory- Note that Vagrantfile creates a CoreOS VM using VirtualBox software hypervisor
- Note that Vagrantfile parses
config.rb
containing a set of options configuring CoreOS cluster
- Generate
user-data
file containing#cloud-config
to simplify provisioning of CoreOS VM using built-incoreos-cloudinit
- Startup and SSH using VirtualBox Provider (default Vagrant provider)
- Note
vagrant up
triggers vagrant to download CoreOS image (if necessary) and relaunch instance - Note
vagrant ssh
connects to VM from directory with Vagrantfile
- Note
Use Vagrant VM:
- Setup shell environment so local Docker client may communicate with Docker daemon on VM
- Show Vagrant environments on host machine
- Check Docker environment variables
- Important: Restart Bash terminal to avoid error connecting to Docker Daemon
- Connect to Docker Daemon (running on port 2375). Note use unencrypted only in Development.
- Important Note: Docker Daemon of Docker for Mac or Docker Toolbox already runs in a Linux VM, so you do not need to (and cannot) run it manually with
dockerd
. It is already running in the whale icon is in the top bar. https://github.com/docker/docker/issues/27102 - Connect to Shell on Vagrant VM
Share Folders between Vagrant VM (CoreOS) and Host (OSX). IDE may now be used to edit files on Vagrant VM.
- Enable synced folders by editing Vagrantfile as follows (uncommenting the
config.vm.synced_folder
line of code)
- Restart Vagrant VM
- Note: If an exports invalid error occurs then open and validate the contents of
/etc/exports
on the Host (OSX) - Check that files synchronise between these folders
Interactive Docker Containers within Vagrant VM (Docker host):
- From the shell of Vagrant VM, test download a Docker Image from Docker Hub for the base distribution specified using Docker Daemon(i.e. Ubuntu, Fedora, and CentOS). A Docker Container inside Vagrant VM filesystem is created based on the downloaded Docker Image.The new Docker Container has a network, IP addresss, and bridge to communicate with localhost.The command and flags run by Docker on the new container launch an interactive Bash shell instance of the host (i.e. Ubuntu) in it that we are logged into as root user.
- List all available Linux commands, aliases, built-ins, keywords, and functions
- Show Docker Container Host Entries
- Show Docker Container IP address
- Show Docker Container Running Processes
- Install software package on Docker Container
- Exit Docker Container to return to Vagrant VM
- List Docker Containers (both stopped and running) from within Vagrant VM. Note: Those that were run with
--rm
flag are not shown.
- List Docker Containers (only last 2 that were stopped or running)
- Create Container (but not Run), Start, Restart, Re-Attach (to process using original Options, i.e. command, flags), and Delete Docker Container
Important Note: Press Enter after Re-Attach to Container (otherwise it hangs)
Daemonized (non-Interactive) Docker Containers within Vagrant VM (Docker host) for running apps and services:
- Create and run a Daemonized Docker Container within Vagrant. Flags detach the container to the background and performs a command until process stopped.
- View under the hood of the Docker Container by debugging the last few log entries
- Monitor (follow) the Docker Container log entries using the
-f
flag (similar to thetail -f
binary) with timestamps and without reading previous entries to the log file
- Note: Transport Layer Security (TLS) is a cryptographic protocol that provides communications security over a network (predecessor SSL)
- Inspect processes (and PIDs) of Daemonized Docker Container
- Statistics (CPU, memory, network, storage I/O, metrics) of Daemonized Docker Container
- Background Processes (using detach
-d
flag) started on already running containers are executed (management, monitoring, or maintenance) with Process Owner in Daemonized Docker Container
- Note:
docker exec
works on already running containers, otherwise error returned:
- Inspect Docker Container (config info, names, commands)
- Selectively Inspect Docker Container (using GoLang template with
-f
flag https://golang.org/pkg/text/template/)
- View Docker Container storage location on Vagrant VM (i.e. CoreOS). Start shell as superuser first
Log Collection
Chapter 2 - Dockerfile, Docker Images
- Docker Container based on a Docker Image (based on a public image or custom) stored in Repository of a Registry
- Docker Image has filesystem layers with mapping to each build step
- Docker Image Management storage backend communicates with underlying Linux filesystem to build layers into usable image
- Docker Image Management storage backends include fast Copy-on-Write (CoW)
- Docker Image Management storage backends supported include AUFS, BTRS, Device-mapper, and overlayfs
- Docker Hub is the default Registry
- Top-Level Repositories are managed by Docker and vendors (i.e.
ubuntu
) - Top-Level Docker Images offer bare runtime to run the distribution
- User Repositories are by individual developers (i.e. ltfschoen/ruby)
- Tag comprises a series of image layers in single repository (representative of version control)
- Docker Image is built upon the Union Mount of its parent image layers above the base image (each layer is a filesystem)
- Show Docker Images downloaded from repositories that contain layers and metadata
- Go to local storage location of Docker Images
- Pull specific Tag (i.e. 3.4) for Top-Level Docker Image
- Search Docker Hub for publicly available Docker Images that may already be pre-installed with applications
- Login/Logout of Docker Hub
Docker Build Command (using Dockerfile)
Docker File
- Dockerfile and
docker build
is repeatable, transparent, and idempotent, and so is preferred overdocker commit
- Dockerfile instructs how Docker Image is to be assembled using application code.
- Dockerfile uses DSL with instructions paired with arguments on each line of code
- Dockerfile lines of code are processed from top to bottom
- Each line of code creates a new Docker Image Layer.
- Only layers deviating from previous Docker Image Layer must be built
- Dockerfile
docker build
process is:1) Docker runs an initial new Docker Container from the base Docker Image (FROM
)2) Docker executes instruction (first line of code) inside initial Docker Container3) Docker runs adocker commit
to commit initial Docker Image Layer4) Docker runs a second new Docker Container based on the initial Docker Image Layer5) Docker executes instruction (next line of code) inside second Docker Container6) Docker runs adocker commit
to commit second Docker Image Layer7) Docker runs a third new Docker Container based on the second Docker Image Layer8) Docker repeats steps 5) to 7) for subsequent lines of code to build up a Docker Container - Docker Container that is built houses the application (i.e. Rails-based app) with all dependencies
- Debug errors when building from Dockerfile by running Docker Container from the last successfully created Docker Image
- Dockerfile
FROM
- Default instance is base Linux image
- Custom Docker Registry offers official framework Docker Images and Tags (i.e. Node.js). Docker Image base inherits from Docker Container defined by
FROM
(i.e.node:0.10.33
specifically locks Docker Image base to specific Ubuntu Linux image running Node 0.10.33)
- Dockerfile
MAINTAINER
- provides author field metadata in Docker Image - Dockerfile
LABEL
- provides KV pairs of metadata in Docker Image for search purposes. Find withdocker inspect
- Dockerfile
USER
- Important: Docker Containers run processes on the host machine kernel. Do not be run as root user in Production for security
- Dockerfile
ENV
- set shell environment variables used in build process on the image - Dockerfile
RUN
- execute instruction/command to append new Docker Image Layer on the current Docker Image
- start and create file structure, and install dependencies
- commit the new Docker Image Layer if command successful
- Important: Reduce build time by abstracting common
RUN
processes (i.e. updates) into code lines (Docker Image layers) of Base Image - Important: Reduce rebuild time by ordering commands with code imports at the end (as all layers after introduced change are rebuilt)
- Important: Combine logically grouped commands into single Dockerfile code line since each instruction creates a new Docker Image layer (i.e. use
ADD
to import scripts and required supporting files from local filesystem into image) - Dockerfile
WORKDIR
- changes working directory in image for remaining build instructions - Dockerfile
CMD
- final instruction launches process to run in Docker Container - Dockerfile
EXPOSE
- ports Docker Container listens to.docker run
flag (--expose
) opens these ports at runtime - Important: Single function in
CMD
to easy horizontal scaling of individual functions in architecture
Build Custom Container
- Build new Custom Container based on Top-Level Container that we build from a Top-Level Image
Example - Static Web Server
- Create build environment (build context) for Docker to upload to Docker Daemon when build is run.
- Note: Previously we configred the Vagantfile to synchronise the following folders between Vagrant VM (CoreOS) and Host (OSX):
~/code/docker-host-coreos-share
(Host)/home/core/share
(Vagrant VM)
- Create Dockerfile that builds a Docker Image where the Docker Container is a static web server
- Note: Perform these on Host machine since folders are synchronised
- Build Dockerfile into Docker Image.Tagging with repository/image_name:version.Use
-f
flag to specify directory within build context containing Dockerfile. - Note:
docker build
must be run on the Vagrant VM (CoreOS), otherwise the following error message appearsCannot connect to the Docker daemon. Is the docker daemon running on this host?
- Check Docker Images updated
- Check Docker Image history (Docker Image Layers are shown)
Docker Image Templates using Build Cache
- Docker uses cache and re-builds from earliest image layer change.Bypass using
--no-cache
flag. - Dockerfile should have template at top of file with package repositories and updates so cache is hit.
Launch Container using Docker Image
-d
flag detaches in background for long processes (i.e. Nginx daemon)-p
flag manages network ports published by Docker at container runtime to the host machine, with options including:- Mapping of randomly assigned Docker Host port(i.e. 32768 to 61000) to Port 80 on Docker Container
- Mapping of specific Docker Host port (i.e. 8080) direct binding to Port 80 on Docker Container(i.e.
sudo docker run -d -p 8080:80 --name static_web ltfschoen/static_web_8080 nginx -g 'daemon off;'
) - Mapping of specific Interface (i.e.
127.0.0.1
) and specific Docker Host port (i.e. 8080) direct binding to Port 80 on Docker Container(i.e.sudo docker run -d -p 127.0.0.1:8080:80 --name static_web ltfschoen/static_web_8080_i127_0_0_1:v1 nginx -g 'daemon off;'
) - Mapping of specific Interface (i.e.
127.0.0.1
) and random Docker Host port to Port 80 on Docker Container(i.e.sudo docker run -d -p 127.0.0.1::80 --name static_web ltfschoen/static_web_random_i127_0_0_1 nginx -g 'daemon off;'
) - Mapping of random Docker Host port to Port 80 on Docker Container and publish additional ports specified with
EXPOSE
instructions(i.e.sudo docker run -d -P --name static_web_all ltfschoen/static_web_all nginx -g 'daemon off;'
) - Note: Random port is assigned when not specified
- Note: Only one Docker Container may be bound to a specific port on host machine
- Note: Bind UDP ports by appending to port binding
/udp
- Command to run (i.e.
nginx -g 'daemon off;'
) launches Nginx in foreground to run web server
- Check Port mapping on host machine assigned to a given port 80 of a Docker Container ID/name
- Selectively Inspect Docker Container
- Connect to the Docker Container housing the Nginx web server using cURL
![Nets docker for mac os Nets docker for mac os](/uploads/1/1/9/8/119875930/502941109.png)
- Attempt with the following:
- Destroy Docker Containers and re-build Docker Image with different Dockerfilewith change to
EXPOSE 80
(not necessary to delete the Docker Container withdocker rmi ltfschoen/static_web:v1
though due to caching)
- Check port 80 is correctly binded by Nginx
- Check IP Table rules that are in place
- Attempt Bridged Adapter option
- Error below when running
vagrant up
after changing from NAT to Bridge Adapter in VirtualBox GUI > Settings > Network > Adapter 1 because Adapter 3 is already set to Bridge Adapter, and making following changes to Vagrantfile:
- Solved with
vagrant halt
, started VM from Virtual Box GUI, then goto bar menu and chose quit VM. https://github.com/mitchellh/vagrant/issues/1809Note: This solution causes another cascading problem: - Error encountered when run
vagrant up
again
- Solved by having both Public and Private in Vagrantfilehttps://github.com/mitchellh/vagrant/issues/2748http://stackoverflow.com/questions/24984020/unable-to-connect-to-vagrant-apache-on-one-computer
- Note: Still unable to get response from VM when curl from Host to Docker Container IP/port (i.e. curl 172.18.0.2:8080)
Misc Links:http://stackoverflow.com/questions/33814696/how-to-connect-to-a-docker-container-from-outside-the-host-same-network-windohttps://www.virtualbox.org/manual/ch06.htmlhttp://stackoverflow.com/questions/25433488/vagrant-can-not-ping-guest-machine-from-the-hosthttp://serverfault.com/questions/495914/vagrant-slow-internet-connection-in-guesthttps://friendsofvagrant.github.io/v1/docs/bridged_networking.html
TODO Default Router approach:http://superuser.com/questions/752954/need-to-do-bridged-adapter-only-in-vagrant-no-nathttps://www.vagrantup.com/docs/networking/public_network.html
TODO Key Link:Alternative using Docker Compose instead of Vagrant https://hharnisc.github.io/
In progress up to CMD section on page 99
Other Links
Other
- Note: Docker Toolbox is for older Macs.
- Uninstall Docker Toolbox. Download shell script and run with
sudo bash uninstall.sh
https://github.com/docker/toolbox/blob/master/osx/uninstall.sh
TODO
- Relevant folders:
- /Users/Ls/code/docker-host-coreos-share/ltfschoen/static_web
- /Users/Ls/code/docker-host/coreos-vagrant
- Learn Chef with Docker
- Control Docker Daemon with:
dockerd
- Apache Kafka (broker) and Apache Zookeeper (run using Docker)
- https://github.com/HackerHappyHour/docker-toolbox-vagrant
- https://docs.docker.com/engine/tutorials/dockerimages/#building-an-image-from-a-dockerfile
- https://docs.docker.com/engine/reference/builder/
- https://www.percona.com/blog/2016/05/11/quick-start-mysql-testing-using-docker-mac/
Introduction
At D2SI, we have been using Docker since its very beginning and have been helping many projects go into production. We believe that going into production requires a strong understanding of the technology to be able to debug complex issues, analyze unexpected behaviors or troubleshoot performance degradations. That is why we have tried to understand as best as we can the technical components used by Docker.
This blog post is focused on the Docker network overlays. The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This article will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers.
This post is derived from the presentation I gave at DockerCon2017 in Austin. The slides are available here.
All the code used in this post is available on GitHub.
Docker Overlay Networks
First, we are going to build an overlay network between Docker hosts. In our example, we will do this with three hosts: two running Docker and one running Consul. Docker will use Consul to store the overlay networks metadata that needs to be shared by all the Docker engines: container IPs, MAC addresses and location. Before Docker 1.12, Docker required an external Key-Value store (Etcd or Consul) to create overlay networks and Docker Swarms (now often referred to as “classic Swarm”). Starting with Docker 1.12, Docker can now rely on an internal Key-Value store to create Swarms and overlay networks (“Swarm mode” or “new swarm”). We chose to use Consul because it allows us to look into the keys stored by Docker and understand better the role of the Key-Value store. We are running Consul on a single node but in a real environment we would need a cluster of at least three nodes for resiliency.
In our example, the servers will have the following IP addresses:
- consul: 10.0.0.5
- docker0: 10.0.0.10
- docker1: 10.0.0.0.11
Starting the Consul and Docker services
The first thing we need to do is to start a Consul server. To do this, we simply download Consul from here. We can then start a very minimal Consul service with the following command:
We use the following flags:
- server: start the consul agent in server mode
- dev: create a standalone Consul server without any persistency
- ui: start a small web interface allowing us to easily look at the keys stored by Docker and their values
- client 0.0.0.0: bind all network interfaces for client access (default is 127.0.0.1 only)
To configure the Docker engines to use Consul as an Key-Value store, we start the daemons with the cluster-store option:
The cluster-advertise option specifies which IP to advertise in the cluster for a docker host (this option is not optional). This command assumes that consul resolves to 10.0.0.5 in our case.
If we look at the at the Consul UI, we can see that Docker created some keys, but the network key: http://consul:8500/v1/kv/docker/network/v1.0/network/ is still empty.
You can easily create the same environment in AWS using the terraform setup in the GitHub repository. All the default configuration (in particular the region to use) is in variables.tf. You will need to give a value to the key_pair variable, either using the command line (terraform apply -var key_pair=demo) or by modifying the variables.tf file. The three instances are configured with userdata: consul and docker are installed and started with the good options, an entry is added to /etc/hosts so consul resolves into the IP address of the consul server. When connecting to consul or docker servers, you should use the public IP addresses (given in terraform outputs) and connect with user “admin” (the terraform setup uses a debian AMI).
Creating an Overlay
We can now create an overlay network between our two Docker nodes:
We are using the overlay driver, and are choosing 192.168.0.0/24 as a subnet for the overlay (this parameter is optional but we want to have addresses very different from the ones on the hosts to simplify the analysis).
Let’s check that we configured our overlay correctly by listing networks on both hosts.
This looks good: both Docker nodes know the demonet network and it has the same id (13fb802253b6) on both hosts.
Let’s now check that our overlay works by creating a container on docker0 and trying to ping it from docker1. On docker0, we create a C0 container, attach it to our overlay, explicitly give it an IP address (192.168.0.100) and make it sleep. On docker1 we create a container attached to the overlay network and running a ping command targeting C0.
We can see that the connectivity between both containers is OK. If we try to ping C0 from docker1, it does not work because docker1 does not know anything about 192.168.0.0/24 which is isolated in the overlay.
Here is what we have built so far:
Under the hood
Now that we have built an overlay let’s try and see what makes it work.
Network configuration of the containers
What is the network configuration of C0 on docker0? We can exec into the container to find out:
We have two interfaces (and the loopback) in the container:
- eth0: configured with an IP in the 192.168.0.0/24 range. This interface is the one in our overlay.
- eth1: configured with an IP in 172.18.0.2/16 range, which we did not configure anywhere
What about the routing configuration?
Nets Docker For Mac Catalina
The routing configuration indicates that the default route is via eth1, which means that this interface can be used to access resources outside of the overlay. We can verify this easily by pinging an external IP address.
Note that it is possible to create an overlay where containers do not have access to external networks using the
--internal
flag.Let’s see if we can get more information on these interfaces:
The type of both interfaces is veth. veth interfaces always always come in pair connected with a virtual wire. The two peered veth can be in different network namespaces which allows traffic to move from one namespace to another. These two veth are used to get outside of the container network namespace.
Here is what we have found out so far:
We now need to identify the interfaces peered with each veth.
What is the container connected to?
We can identify the other end of a veth using the ethtool command. However this command is not available in our container. We can execute this command inside our container using “nsenter” which allows us to enter one or several namespaces associated with a process or using “ip netns exec” which relies on iproute to execute a command in a given network namespace. Docker does not create symlinks in the /var/run/netns directory which is where ip netns is looking for network namespaces. This is why we will rely on nsenter for namespaces created by Docker.
To list the network namespaces created by Docker we can simply run:
To use this information, we need to identify the network namespace of containers. We can achieve this by inspecting them, and extracting what we need from the SandboxKey:
We can also execute host commands inside the network namespace of a container (even if this container does not have the command):
Let’s see what are the interface indexes associated with the peers of eth0 and eth1:
We are now looking for interfaces with indexes 7 and 10. We can first look on the host itself:
We can see from this output that we have no trace of interface 7 but we have found interface 10, the peer of eth1. In addition, this interface is plugged on a bridge called “docker_gwbridge”. What is this bridge? If we list the networks managed by docker, we can see that it has appeared in the list:
We can now inspect it:
I removed part of the output to focus on the essential pieces of information:
- this network uses the driver bridge (the same one used by the standard docker bridge, docker0)
- it uses subnet 172.18.0.0/16, which is consistent with eth1
- enable_icc is set to false which means we cannot use this bridge for inter-container communication
- enable_ip_masquerade is set to true, which means the traffic from the container will be NATed to access external networks (which we saw earlier when we successfully pinged 8.8.8.8)
We can verify that inter-container communication is disabled by trying to ping C0 on its eth1 address (172.18.0.2) from another container on docker0 also attached to demonet:
Here is an updated view of what we have found:
What about eth0, the interface connected to the overlay?
The interface peered with eth0 is not in the host network namespace. It must be in another one. If we look again at the network namespaces:
We can see a namespace called “1-13fb802253”. Except for the “1-“, the name of this namespace is the beginning of the network id of our overlay network:
This namespace is clearly related to our overlay network. We can look at the interfaces present in that namespace:
The overlay network namespace contains three interfaces (and lo):
Nets Docker For Mac Os
- br0: a bridge
- veth2: a veth interface which is the peer interface of eth0 in our container and which is connected to the bridge
- vxlan0: an interface of type “vxlan” which is also connected to the bridge
The vxlan interface is clearly where the “overlay magic” is happening and we are going to look at it in details but let’s update our diagram first:
Conclusion
This concludes part 1 of this article. In part 2, we will focus on VXLAN: what is this protocol and how it is used by Docker.