Deploy AI Runtime Security Container on AWS

These installation steps show you how to install locally into a Minikube cluster. Instructions are also valid for Kubernetes provided by Docker Desktop and a full-fledged Kubernetes cluster. Running locally with Minikube introduces extra latency due to the prompt injection classification model. Latency is not present in a full-fledged Kubernetes cluster.

Before deploying Model Scanner, make sure you understand:

Select OS

Select your operating system to view installation instructions.

macOS

Windows

Ubuntu

Open a terminal and create an environment variable file with Quay credentials. Use the following examples. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.
- For Quay credentials and other requirements, see Resource Requirements.
```
cat << EOF > env
HIDDENLAYER_USERNAME=%QUAY_USERNAME%
HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
EOF
```

Special Characters

Special characters will require wrapping the value in a string.

Load the environment variables into your shell.
```
source ./env
```

Select Deployment Type

Select your deployment type to view instructions.

Hybrid

Disconnected

A Hybrid deployment sends metadata to the HiddenLayer AI Security Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

Create a file named .env.local.

Use the comments in the example to find configurations for your deployment needs.
File changes to note:
- For the EU region, uncomment hl_region=eu.
- Replace <license> with your Runtime Security license.
- Replace <client id> and <client secret> with your HiddenLayer API key and secret.
- Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.

cat << EOF > .env.local
namespace:
  name=aidr-genai
image:
  ## By default, the image below is deployed; this image is for CPU deployments
  repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
  ## To set up a GPU deployment, uncomment the line below to use the CUDA image
  # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
  ## By default, the latest version is installed
  ## Uncomment the following to specify a specific version
  # tag=25.5.2
## By default, hybrid connection will connect to the 'US' region
## Uncomment the following option to change to a different supported region  
# hl_region=eu
resources:
  ## For CPU deployments, 8 vCPUs are recommended
  ## For GPU deployments, 4 vCPUs are recommended
  requests:
    cpu=8
  ## For GPU deployments, uncomment this to allocate 1 GPU per instance
  # limits:
    # nvidia.com/gpu=1
## Set number of instances for deployment
replicas:
  min=1
  max=1
## Specify a specific node group if necessary
## For instance, GPU deployments may require specific nodes to access a GPU
# nodeGroup=<nodegroup_name>
config:
  HL_LICENSE=<license>
  HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
  HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>
  ## This must match the number of vCPUs assigned above
  OMP_NUM_THREADS=8
## By default, the hybrid connection type is used and requires a client id and secret
  HL_LLM_PROXY_CLIENT_ID=<client_id>
  HL_LLM_PROXY_CLIENT_SECRET=<client_secret>
  ## Uncomment the following to disable the connection to the AI Security Platform
  ## in this mode, the client id and secret are not required
  # HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
EOF

Policy Configuration Environment Variables

For other policy configuration environment variables, see Runtime Security Configuration.

Make sure Docker Desktop is running.

Run the following command to login to the HiddenLayer Quay repository.

docker login --username=$HIDDENLAYER_USERNAME --password=$HIDDENLAYER_PASSWORD quay.io

Run the Runtime Security container.

docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest

Health Check

To check that the proxy is up and running, open a new terminal and use the following command to ping the health monitor.

curl http://localhost:8000/health

Generate Manifest

To write the generated manifest to a file (manifest.yml), open a terminal and run the following command.

You must run the Runtime Security installer before generating a manifest.
Change latest to the Runtime Security version that you use.

docker run --env-file ./env -v pwd/config:/src/config quay.io/hiddenlayer/distro-enterprise-aidr-genai-installer:latest > manifest.yml