Before we jump into instrumenting Azure Container Instances(ACI) with Datadog APM, I will briefly cover achieving the three pillars of observability for ACI through Datadog APM.

Three pillars of observability

The three pillars of observability including the 3 vital data for a complete visibility of your application. These are metrics, logs and traces.

Metrics - Metrics measure the various aspects of the application and infrastructure as a quantitative measure. The Datadog integration for Microsoft Azure can be used to collect the metrics specific to Azure Container Instances.
Logs - Logs contain the information emitted by the application during runtime and is printed by the developers to a log stream. Datadog collects logs from containers using the container label com.datadoghq.ad.logs. This approach will not work for ACI since the process requires access to the docker socket and there is no way to mount the docker socket onto an ACI container.
Traces - Traces represent an entire execution path of a request/transaction. I will cover how to collect application traces for a Java application through Datadog's APM agent in this blog.

Instrumentation of Azure Container Instances with Datadog

Datadog requires two kinds of agents for collecting APM data - APM agent and host agent. The APM agent is associated with the application runtime and the host agent runs on the host and is responsible for other Datadog functionalities such as collecting metrics, logs and forwarding other data to the Datadog server.

Azure Container Instances allow a group of containers to be run in a single pod(container group) and these containers are run in the same networking and other namespaces, similar to how pods function in Kubernetes. Since the host of an ACI is abstracted, we will run the Datadog host agent as a sidecar container along with the application container in the same group.

We will be using the Azure CLI to interact with Azure for two reasons -

Easier to automate the process
An Azure Container group with multiple containers can only be created using a YAML file or an Infrastructure as Code(IaC) tool and the YAML file can be passed to Azure using the Azure CLI.

Prerequisites

Some prerequisites to follow along -

Datadog account (Datadog offers a free 14 day trial)
Microsoft Azure account (Azure offers a free trial as well)
Azure CLI installed in a Linux environment and preferably in bash. Login to your Azure account from the CLI using az login and set a default subscription for the CLI using az account set

Setup

Architecture

Once the prerequisites are met, we will proceed with the creation of Azure resources starting with a Resource Group. Resource Groups are logical collections of all related resources in Azure. A handy feature of Azure Resource Groups that are helpful for scenarios such as PoCs are that Azure deletes all the resources in a Resource Group if you delete the Resource Group. You will not have to worry about cleaning up or a huge cloud bill 😁.

Before we create the Resource Group, let us define some environment variables that will be helpful as we create more Azure resources.

export AZ_LOCATION=centralindia
export AZ_RESOURCE_GROUP=aci-datadog-apm
export AZ_STORAGE_ACCOUNT=apmagents
export AZ_STORAGE_SHARE=datadog-apm

Now, let us create the Resource Group aci-datadog-apm in the location centralindia -

az group create --location $AZ_LOCATION --name $AZ_RESOURCE_GROUP

We will use an Azure File Share to store the Datadog APM agent and mount the File Share onto the ACI to provide the agent to the application. You could provide the APM agent through the Dockerfile as well by copying the agent into the Dockerfile. I prefer using a File Share since that allows me to update the agent version without rebuilding my application container.

In order to create an Azure File Share, we will require an Azure Storage Account, which holds all of Azure's Storage resources. We can create one using -

az storage account create --name $AZ_STORAGE_ACCOUNT \
    --resource-group $AZ_RESOURCE_GROUP \
    --sku Standard_LRS \
    --location $AZ_LOCATION

Once the storage account is created, we will require the storage account's key to interact with the storage account. We can fetch the Storage Account Key using -

export AZ_STORAGE_ACCOUNT_KEY=`az storage account keys list -g $AZ_RESOURCE_GROUP -n $AZ_STORAGE_ACCOUNT | jq -r '.[1].value'`

Note: You will require the jq tool for the previous step. If you do not have/want to install jq, you can run az storage account keys list -g $AZ_RESOURCE_GROUP -n $AZ_STORAGE_ACCOUNT and copy one of the two keys and assign it to a variable called AZ_STORAGE_ACCOUNT_KEY

Let us create the Azure File Share by running -

az storage share create --name $AZ_STORAGE_SHARE \
    --account-key $AZ_STORAGE_ACCOUNT_KEY \
    --account-name $AZ_STORAGE_ACCOUNT

We need to then download the Datadog APM agent locally and upload it to the Azure File Share.

# Download the Datadog APM agent locally
wget -O dd-java-agent.jar https://dtdg.co/latest-java-tracer

# Upload Datadog APM agent to the file share
az storage file upload --account-name $AZ_STORAGE_ACCOUNT \
    --account-key $AZ_STORAGE_ACCOUNT_KEY \
    --share-name $AZ_STORAGE_SHARE \
    --source dd-java-agent.jar

We have the necessary resources setup except for the Azure Container Instance group. Let us take a look at the Docker file before we create the Azure Container Instance Group -

FROM maven:3.8.4-jdk-11-slim as build
COPY . /usr/src/app/
RUN mvn -f /usr/src/app/pom.xml clean install -DskipTests

FROM gcr.io/distroless/java
EXPOSE 8080
COPY --from=build /usr/src/app/target/*.jar /usr/app/*.jar
ENTRYPOINT ["java", "-javaagent:/opt/datadog/dd-java-agent.jar", "-jar", "/usr/app/*.jar"]

Notice that the final line of the Dockerfile specifies the path to the APM agent. We need to ensure the APM agent is mounted at /opt/datadog.

The final step is to create the Azure Container Instance group using the YAML file. The definition for the file is as follows -

name: aci-datadog
apiVersion: '2021-10-01'
location: centralindia
properties: 
  containers: 
  - name: hello-world
    properties: 
      image: avinashupadhya99/hello-world-spring-boot-datadog
      ports: 
      - protocol: TCP
        port: 8080
      environmentVariables:
      - name: DD_ENV
        value: az-test
      - name: DD_SERVICE
        value: hello-world-aci
      - name: DD_VERSION
        value: "0.1.0"
      volumeMounts:
      - name: datadog-agent
        mountPath: /opt/datadog/
      resources:
        requests:
          memoryInGB: 0.3
          cpu: 0.2
        limits:
          memoryInGB: 0.3
          cpu: 0.2
  - name: datadog-agent
    properties: 
      image: datadog/agent
      environmentVariables:
      - name: DD_API_KEY
        secureValue: DATADOG_API_KEY # To be replaced before applying
      - name: DD_SITE
        value: "datadoghq.com"
      - name: DD_ENV
        value: az-test
      - name: DD_SERVICE
        value: hello-world-aci
      - name: DD_APM_ENABLED
        value: "true"
      resources:
        requests:
          memoryInGB: 0.3
          cpu: 0.2
        limits:
          memoryInGB: 0.3
          cpu: 0.2
  ipAddress:
    ports:
    - protocol: TCP
      port: 8080
    type: Public
    dnsNameLabel: aci-datadog-apm # Ensure this is globally unique per location
  osType: Linux
  volumes:
  - name: datadog-agent
    azureFile:
      shareName: datadog-apm
      readOnly: true
      storageAccountName: apmagents
      storageAccountKey: STORAGE_ACCOUNT_KEY # To be replaced before applying

Save the contents of the YAML definition in a file called azure-container-group.yaml

We have defined the three unified tags of Datadog - env, service and version as environment variables for the application container(hello-world). A volume called datadog-agent is defined using azureFile and is mounted to the application container at /opt/datadog. This allows the application container to use the APM agent as defined in the Dockerfile. Ensure that the value for each of the parameters under azureFile are the same as the Azure Storage Account and File Share that was created earlier.

The Datadog host agent is defined in the container datadog-agent and it requires your Account API key to send the collected traces to your Datadog account. You can find the instructions to create your API key here. Store the API key in a variable called DD_API_KEY as

# Replace with your Datadog API key
export DD_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The rest of the environment variables are not strictly necessary and I have mentioned it for being more verbose. The Datadog host agent is configured through the various environment variables that are provided to the Datadog agent container.

Before, we apply the YAML file and create the Azure Container Instance, let us replace the placeholders DATADOG_API_KEY and STORAGE_ACCOUNT_KEY since these are secret values and the Azure CLI does not support environment variable interpolation or other ways of providing the value securely.

sed 's/DATADOG_API_KEY/$DD_API_KEY/g' azure-container-group.yaml > azure-container-group-with-values.yaml
sed 's/STORAGE_ACCOUNT_KEY/$STORAGE_ACCOUNT_KEY/g' azure-container-group-with-values.yaml > azure-container-group-with-values.yaml

Ensure that the file azure-container-group-with-values.yaml is not shared or committed to a version control since the file contains secret values that should not be shared.

Let us create the Azure Container Instance group using -

az container create --file azure-container-group-with-values.yaml \
    --resource-group $AZ_RESOURCE_GROUP

Once the container is created, let us generate some traffic before we check the traces on Datadog. Note down the FQDN from the output of the previous command. It will be in the following format - <DNS_LABEL>.<LOCATION>.azurecontainer.ioand in my case it is aci-datadog-apm.centralindia.azurecontainer.io. We will generate the traffic by calling the home route 5 times -

for i in {1..5}
do
   curl http://aci-datadog-apm.centralindia.azurecontainer.io:8080/
done

Finally, let us verify if we are able to capture the traces in Datadog by visiting app.datadoghq.com/apm/traces?query=env%3Aaz.. Notice the traces and spans in the Datadog UI -

Datadog UI with traces

Click on any of the spans to know more about the request or click on the Services on the top to learn more about the performance of the service.

We were successfully able to instrument Azure Container Instances with Datadog APM and view the traces in the Datadog UI.

Instrumenting Azure Container Instances with Datadog APM

Setting up Observability for Azure Container Instances with Datadog

Three pillars of observability

Instrumentation of Azure Container Instances with Datadog

Prerequisites

Setup