What is Docker?
Docker is an open-source platform that allows you to develop, deploy, and manage multiple applications across one operating system. Instead of having many separate servers all configured and dedicated to a specific application, you can run all of your applications from one server, with each application in its own container. For more information on what Docker is, and how it works, check out their technical documentation.
How hackers currently scale automation
Currently, many hackers deploy scripts and applications to virtual private servers (VPSs). As you can probably imagine, running multiple scripts on multiple VPSs quickly leads to a very difficult automation workflow. As an example, you may have three small servers running subdomain enumeration, three running port scans, three running nuclei, and one running a central database. This leaves 10 servers in total.
The only way you can scale up is by adding more servers. Every server adds more operational overheads.
Docker automation scaling
Automation for ethical hacking is a great use case for microservices. Scaling automation with Docker is approaching automation scaling with microservice architecture. Microservice architecture is building a system of small services that operate independently to achieve a common goal. Each step of your workflow can be an isolated microservice with its own Docker image and container. Multiple Docker containers can be deployed onto a single VPS which makes scaling as easy as deploying more containers.
Let’s look at an example of building a microservice architecture using Docker.
For this example, we have a database to store our data, a Redis queue for distributing tasks, and a VPS to host our Docker containers. Cron jobs push tasks to the queue until a container is available to run them. We have 3 containers: one for passive subdomain enumeration, one for domain resolving, and one for discovering HTTP servers. The number of containers can be scaled up or down to meet the needs of the Redis queue. When the tasks are completed, results are stored in the database. To scale up, we simply add more containers.
Making things even easier with Docker Compose
Docker Compose is a tool for running multi-container applications. It uses YAML files to define the configuration of the containers. Once the file is created, all you will need to do is run “docker-compose up”. To scale up, just increase the number of containers in the YAML file, and Docker Compose does the rest. Here’s what the Docker Compose file looks like.
How to scale automation
To show how you can scale automation, let’s make a single container environment and install some tools within that container. We will use the following:
- A Redis server from Digital Ocean.
- A VPS to run containers.
- A server with hakscale configured to push subfinder commands.
Hakscale pushes commands to our Redis queue and pops them to subfinder. You need three files in a directory to get set up:
- Dockerfile
- docker-compose.yaml
- hakscale-config.yaml
Dockerfile
FROM golang:1.19.2-alpine
# Adding git to install tools!
RUN apk update && apk add bash git
# Copy hakscale config file with Redis queue info!
COPY hakscale-config.yaml /root/.config/haktools/hakscale-config.yml
# Install hakscale from hakluke!
RUN go install github.com/hakluke/hakscale@latest
# Install subfinder from project discovery!
RUN go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
ENTRYPOINT hakscale pop -q blogDemo
docker-compose.yaml
version: “3.9”
services:
subdomain_worker:
image: subdomain_worker
build:
context: .
dockerfile: Dockerfile
network_mode: host
hakscale-config.yaml
redis:
host:
Once you have your Redis queue deployed, the three files created in a directory, and hakscale configured, you can spin up your example worker by running `docker-compose up`. Once the build is complete you can push commands to it.
Next, create a file of domains you would like to test, for example:
$ cat testdomains.txt
bugcrowd.com
hackerone.com
Run `hakscale push -p “host:./testdomains.txt” -c “subfinder -d _host_” -t 20 -q blogDemo`. This will push two commands to the queue. The worker will pick them up and start processing them. When it finishes it should print the output back to the queue, similar to below:
Worker output
Queue ouput
The worker will continue to run jobs from the queue until none are left. It will then poll until there are more to run. To scale the automation quickly and efficiently using this containerized method, you can easily spin up as many of these containers as required.
Conclusion
I hope this guide has convinced you to give Docker a shot when setting up your automation infrastructure. Docker can initially seem intimidating, but I promise that this will improve your automation’s development and deployment. I use this architecture personally for my automation, so feel free to reach out to me personally for advice.
Written by:
Gunnar Andrews
My online alias is G0lden. I am a hacker out of the midwest United States. I came into the hacking world through corporate jobs out of college, and I also do bug bounties. I enjoy finding new ways to hunt bugs and cutting-edge new tools. Making new connections with fellow hackers is the best part of this community for me!