Configuring Chalk Reports to Snapshot Network Connections on Heartbeat

Version

1.0.0

TLDR

This document is a guide on how to configure chalk so that a chalked binary or docker container emits a snapshot of network connections at set intervals.

Prerequisites

chalk binary
(optional) dockerfile for a docker image with compatible architecture

Related Docs and References

The sample configuration file used for reference is included below:

# exec heartbeat
exec.heartbeat: true
exec.heartbeat_rate: <<10 seconds>>

# docker wrapping
docker.wrap_entrypoint: true

# network reporting template
report_template network_report {
  key.CHALK_ID.use                            = true
  key.CHALK_PTR.use                           = true

  key._OPERATION.use                          = true
  key._TIMESTAMP.use                          = true

  key._CHALKS.use                             = true

  key._OP_PLATFORM.use                        = true
  key._OP_HOSTNAME.use                        = true
  key._OP_HOSTINFO.use                        = true
  key._OP_NODENAME.use                        = true

  key._OP_ERRORS.use                          = true

  key._OP_TCP_SOCKET_INFO.use                 = true
  key._OP_UDP_SOCKET_INFO.use                 = true
  key._OP_IPV4_ROUTES.use                     = true
  key._OP_IPV6_ROUTES.use                     = true
  key._OP_IPV4_INTERFACES.use                 = true
  key._OP_IPV6_INTERFACES.use                 = true
  key._OP_ARP_TABLE.use                       = true
}


# output sinks
sink_config network_std_out {
  sink: "stdout"
  enabled: true
}

sink_config network_file_out {
  sink: "file"
  enabled: true
  filename: "~/network_heartbeat_log.log"
}

# custom reporting
custom_report network_heartbeat_report {
  enabled: true
  report_template: "network_report"
  sink_configs: ["network_std_out", "network_file_out"]
  use_when: ["heartbeat"]
}

Save this config as guide_heartbeat.c4m where chalk has access to it..

Steps

Step 1: Enabling Heartbeat Via Config

Exec Heartbeat

The heartbeat functionality on the chalk exec command allows the user to set a heartbeat that will emit a chalk report at the specified interval as long as the target of the exec command is running. The target will not be affected by any report generation or output and should run as normal.

To enable heartbeat and set a heartbeat rate, add the following to the chalk config file:

exec.heartbeat: true
exec.heartbeat_rate: <<10 minutes>>

With this configuration, heartbeat will output a chalk report every 10 minutes.

Docker Wrapping (Optional)

If the target of the chalk exec command is intended to be run via docker, the docker image must be built by chalk with entrypoint wrapping enabled.

To enable docker entrypoint wrapping, add the following to the chalk config file:

docker.wrap_entrypoint: true

Then build the Dockerfile with chalk docker build.

For docker entrypoint wrapping to work, chalk will build the docker image with a copy of the chalk binary inside the image that will then run chalk exec on the original docker entrypoint. By default, the chalk running chalk docker build will attempt to copy itself into the docker image, but will not do so if it detects an architecture mismatch (ex, if the chalk binary was built for linux/arm64 but the target docker image is linux/amd64). In this case, chalk will not be available inside the container and no reports will be generated, although this will not otherwise affect the running of the container.

If you have a compatible chalk binary available for the target container architecture, this chalk can be passed into the build by adding the following line into the config file:

docker.arch_binary_locations: {"linux/amd64": "/path/to/amd/chalk"}

Note that the chalk binary at /path/to/amd/chalk may not be the same as the chalk binary in chalk docker build. The binary being passed via docker.arch_binary_locations must have the heartbeat configuration loaded for the reports to be generated by the running container.

Step 2: Setting Up A Network Reporting Template

Now that heartbeat is enabled, a chalk report will be generated on every heartbeat. To include network data in these reports without including other data that we don't care about, we can define custom reporting templates with just the keys that we want.

Report Template

Our reporting template will include some basic data about the system, as well as the networking-specific data we are looking for. We define it in the config file as follows:

# network reporting template
report_template network_report {
  key.CHALK_ID.use                            = true
  key.CHALK_PTR.use                           = true
  key._OPERATION.use                          = true
  key._TIMESTAMP.use                          = true

  key._CHALKS.use                             = true
  key._OP_PLATFORM.use                        = true
  key._OP_HOSTNAME.use                        = true
  key._OP_HOSTINFO.use                        = true
  key._OP_NODENAME.use                        = true
  key._OP_ERRORS.use                          = true
  key._OP_TCP_SOCKET_INFO.use                 = true
  key._OP_UDP_SOCKET_INFO.use                 = true
  key._OP_IPV4_ROUTES.use                     = true
  key._OP_IPV6_ROUTES.use                     = true
  key._OP_IPV4_INTERFACES.use                 = true
  key._OP_IPV6_INTERFACES.use                 = true
  key._OP_ARP_TABLE.use                       = true
}

Let's examine the reporting keys that we have enabled in this profile:

CHALK_ID and CHALK_PTR are useful to have here to link chalked artifacts
_OPERATION in this case will always be heartbeat for heartbeat reports
_TIMESTAMP is the time when the heartbeat report was generated
_CHALKS is necessary to display the artifact data (ex: CHALK_ID)
_OP_PLATFORM, _OP_HOSTNAME, _OP_HOSTINFO, and _OP_NODENAME provide context about the host system on which chalk is running
_OP_ERRORS reports any errors that may have occurred during the heartbeat operation (this field will not be present in the chalk report if there are no errors)

The other keys provide the networking data for the host on which chalk is running. Other data (such as process information) is not included in this report, but can easily be added by enabling them in the profile defined above -- a full list of available keys can found in src/configs/base_keyspecs.c4m.

Step 3: Defining Output Sinks

To separate our networking reports from normal chalk logs, we can specify custom output sinks where we would like them to be sent. Let's say that we want to print our network reports to terminal, as well as write them to a file for later analysis. We can add the following to our config file:

sink_config network_std_out {
  sink: "stdout"
  enabled: true
}
sink_config network_file_out {
  sink: "file"
  enabled: true
  filename: "~/network_heartbeat_log.log"
}

Here we have defined two sink configurations:

network_std_out sends output to the stdout sink
network_file_out sends output to a file sink, in this case the log file ~/network_heartbeat_log.log. If the file does not exist, it will be created.

Note that if we are running chalk inside a docker container, the reports going to network_file_out will be written to a file INSIDE the running docker container, and will not be accessible from the host. The stdout output will be available via docker logs.

Step 4: Defining A Custom Report

Now that the report template and sink outputs are defined, we can enable them on heartbeat events by creating a custom report that uses these components and adding it to our config file:

custom_report network_heartbeat_report {
  enabled: true
  report_template: "network_report"
  sink_configs: ["network_std_out", "network_file_out"]
  use_when: ["heartbeat"]
}

The fields in the custom report network_heartbeat_report are:

enabled to toggle the report on or off
report_template takes the report template that we want to display, in this case the network_report
sink_configs takes an array of sink outputs that we want to report to, in this case to stdout and to the file we have specified in the network_file_out definition
use_when takes an array of the chalk operations on which we want to report with these profiles. Since ONLY want to generate these network information reports on heartbeat operations, and not on other chalk operations such as exec or extract, we pass in only heartbeat here.

Note that this custom report takes a report_template, but not a mark_template -- this is because heartbeat doesn't generate any chalk marks, so a chalk mark template is not needed here.

Also note that this custom report will not interfere with any default reporting that have been set up. The default reports for chalk insert, chalk extract, and other operations will still be sent to the default sinks, with the data specified by the keys as per the default profiles.

Step 5: Running Chalk

We can test our newly created configuration file by running chalk exec on a target binary and checking that the resulting heartbeat reports contain the expected networking information.

First, we load our configuration into the chalk binary:

chalk load guide_heartbeat.c4m

Then we run chalk exec on a target binary. For testing purposes, we will use sleep:

chalk exec --exec-command-name=sleep 5000

After 10 minutes, we should expect a chalk report on the heartbeat operation that looks similar to this:

[
    {
        "_OPERATION": "heartbeat",
        "_TIMESTAMP": 1694879737530,
        "_CHALKS": [
            {
                "CHALK_ID": "EW3D60-MK8X-ZEHH-DJWE0S"
            }
        ],
        "_OP_HOSTINFO": "#32~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 18 10:40:13 UTC 2",
        "_OP_HOSTNAME": "c687c21b8106",
        "_OP_IPV4_ROUTES": [
            [
                "0.0.0.0",
                "172.17.0.1",
                "0.0.0.0",
                "eth0",
                "0003",
                "0",
                "0",
                "0",
                "0",
                "0",
                "0"
            ],
            [
                "172.17.0.0",
                "0.0.0.0",
                "255.255.0.0",
                "eth0",
                "0001",
                "0",
                "0",
                "0",
                "0",
                "0",
                "0"
            ]
        ],
        "_OP_IPV6_ROUTES": [
            [
                "0000:0000:0000:0000:0000:0000:0000:0000",
                "00",
                "0000:0000:0000:0000:0000:0000:0000:0000",
                "00",
                "0000:0000:0000:0000:0000:0000:0000:0000",
                "lo",
                "00200200",
                "00000001",
                "00000000",
                "ffffffff"
            ]
        ],
        "_OP_NODENAME": "c687c21b8106",
        "_OP_PLATFORM": "GNU/Linux x86_64"
    }
]

This report will be printed to stdout, and checking the log file we have specified at ~/network_heartbeat_log.log should have the same report.

Running Chalk Inside Docker (Optional)

Setting up heartbeat reports is slightly different for docker.

First, we have to load the configuration file into our chalk binary:

chalk load guide_heartbeat.c4m

Then, we have to build the target dockerfile with chalk:

chalk docker build -t [tagname] -f [path/to/dockerfile]

Run the resulting docker image as usual:

docker run [tagname]

The docker logs can be checked via

docker logs [container-name]

which should have the same chalk report on the heartbeat operation as above. Alternatively, you can shell into the running docker container and examine the file ~/network_heartbeat_log.log to see the reports.

Known Issues IDs and Hashing in Chalk