This is the multi-page printable view of this section. Click here to print.
Configure UDS Core
1 - Monitoring and Metrics
UDS Core leverages Pepr to handle setup of Prometheus scraping metrics endpoints, with the particular configuration necessary to work in a STRICT mTLS (Istio) environment. We handle this with both mutations of existing service monitors and generation of service monitors via the Package
CR.
Mutations
All service monitors are mutated to set the scrape scheme to HTTPS and set the TLS Config to what is required for Istio mTLS scraping (see this doc for details). Beyond this, no other fields are mutated. Supporting existing service monitors is useful since some charts include service monitors by default with more advanced configurations, and it is in our best interest to enable those and use them where possible.
Assumptions are made about STRICT mTLS here for simplicity, based on the istio-injection
namespace label. Without making these assumptions we would need to query PeerAuthentication
resources or another resource to determine the exact workload mTLS posture.
Note: This mutation is the default behavior for all service monitors but can be skipped using the annotation key uds/skip-sm-mutate
(with any value). Skipping this mutation should only be done if your service exposes metrics on a PERMISSIVE mTLS port.
Package CR monitor
field
UDS Core also supports generating service monitors from the monitor
list in the Package
spec. Charts do not always support service monitors, so generating them can be useful. This also provides a simplified way for other users to create service monitors, similar to the way we handle VirtualServices
today. A full example of this can be seen below:
...
spec:
monitor:
- selector: # Selector for the service to monitor
app: foobar
portName: metrics # Name of the port to monitor
targetPort: 1234 # Corresponding target port on the pod/container (for network policy)
# Optional properties depending on your application
description: "Metrics" # Add to customize the service monitor name
podSelector: # Add if pod labels are different than `selector` (for network policy)
app: barfoo
path: "/mymetrics" # Add if metrics are exposed on a different path than "/metrics"
This config is used to generate service monitors and corresponding network policies to setup scraping for your applications. The ServiceMonitor
s will go through the mutation process to add tlsConfig
and scheme
to work in an istio environment.
This spec intentionally does not support all options available with a ServiceMonitor
. While we may add additional fields in the future, we do not want to simply rebuild the ServiceMonitor
spec since mutations are already available to handle Istio specifics. The current subset of spec options is based on the bare minimum necessary to craft resources.
NOTE: While this is a rather verbose spec, each of the above fields are strictly required to craft the necessary service monitor and network policy resources.
Notes on Alternative Approaches
In coming up with this feature a few alternative approaches were considered but not chosen due to issues with each one. The current spec provides the best balance of a simplified interface compared to the ServiceMonitor
spec, and a faster/easier reconciliation loop.
Generation based on service lookup
An alternative spec option would use the service name instead of selectors/port name. The service name could then be used to lookup the corresponding service and get the necessary selectors/port name (based on numerical port). There are however 2 issues with this route:
- There is a timing issue if the
Package
CR is applied to the cluster before the app chart itself (which is the norm with our UDS Packages). The service would not exist at the time thePackage
is reconciled. We could lean into eventual consistency here, if we implemented a retry mechanism for thePackage
, which would mitigate this issue. - We would need an “alert” mechanism (watch) to notify us when the service(s) are updated, to roll the corresponding updates to network policies and service monitors. While this is doable it feels like unnecessary complexity compared to other options.
Generation of service + monitor
Another alternative approach would be to use a pod selector and port only. We would then generate both a service and servicemonitor, giving us full control of the port names and selectors. This seems like a viable path, but does add an extra resource for us to generate and manage. There could be unknown side effects of generating services that could clash with other services (particularly with istio endpoints). This would otherwise be a relative straightforward approach and is worth evaluating again if we want to simplify the spec later on.
2 - UDS Operator
The UDS Operator plays a pivotal role in managing the lifecycle of UDS Package Custom Resources (CRs) along with their associated resources like NetworkPolicies and Istio VirtualServices. Leveraging Pepr, the operator binds watch operations to the enqueue and reconciler, taking on several key responsibilities for UDS Packages and exemptions:
Package
- Enabling Istio Sidecar Injection:
- The operator facilitates the activation of Istio sidecar injection within namespaces where the CR is deployed.
- Establishing Default-Deny Ingress/Egress Network Policies:
- It sets up default-deny network policies for both ingress and egress, creating a foundational security posture.
- Implementing Layered Allow-List Approach:
- A layered allow-list approach is applied on top of default-deny network policies. This includes essential defaults like Istio requirements and DNS egress.
- Providing Targeted Remote Endpoints Network Policies:
- The operator creates targeted network policies for remote endpoints, such as
KubeAPI
andCloudMetadata
. This approach aims to enhance policy management by reducing redundancy (DRY) and facilitating dynamic bindings in scenarios where static definitions are impractical.
- The operator creates targeted network policies for remote endpoints, such as
- Creating Istio Virtual Services and Related Ingress Gateway Network Policies:
- In addition, the operator is responsible for generating Istio Virtual Services and the associated network policies for the ingress gateway.
Example UDS Package CR
apiVersion: uds.dev/v1alpha1
kind: Package
metadata:
name: grafana
namespace: grafana
spec:
network:
# Expose rules generate Istio VirtualServices and related network policies
expose:
- service: grafana
selector:
app.kubernetes.io/name: grafana
host: grafana
gateway: admin
port: 80
targetPort: 3000
# Allow rules generate NetworkPolicies
allow:
- direction: Egress
selector:
app.kubernetes.io/name: grafana
remoteGenerated: Anywhere
- direction: Egress
remoteNamespace: tempo
remoteSelector:
app.kubernetes.io/name: tempo
port: 9411
description: "Tempo"
# SSO allows for the creation of Keycloak clients and with automatic secret generation
sso:
- name: Grafana Dashboard
clientId: uds-core-admin-grafana
redirectUris:
- "https://grafana.admin.uds.dev/login/generic_oauth"
Exemption
- Exemption Scope:
- Granting exemption for custom resources is restricted to the
uds-policy-exemptions
namespace by default, unless specifically configured to allow exemptions across all namespaces.
- Granting exemption for custom resources is restricted to the
- Policy Updates:
- Updating the policies Pepr store with registered exemptions.
Example UDS Exemption CR
apiVersion: uds.dev/v1alpha1
kind: Exemption
metadata:
name: neuvector
namespace: uds-policy-exemptions
spec:
exemptions:
- policies:
- DisallowHostNamespaces
- DisallowPrivileged
- RequireNonRootUser
- DropAllCapabilities
- RestrictHostPathWrite
- RestrictVolumeTypes
matcher:
namespace: neuvector
name: "^neuvector-enforcer-pod.*"
- policies:
- DisallowPrivileged
- RequireNonRootUser
- DropAllCapabilities
- RestrictHostPathWrite
- RestrictVolumeTypes
matcher:
namespace: neuvector
name: "^neuvector-controller-pod.*"
- policies:
- DropAllCapabilities
matcher:
namespace: neuvector
name: "^neuvector-prometheus-exporter-pod.*"
Example UDS Package CR with SSO Templating
By default, UDS generates a secret for the Single Sign-On (SSO) client that encapsulates all client contents as an opaque secret. In this setup, each key within the secret corresponds to its own environment variable or file, based on the method used to mount the secret. If customization of the secret rendering is required, basic templating can be achieved using the secretTemplate
property. Below are examples showing this functionality. To see how templating works, please see the Regex website.
apiVersion: uds.dev/v1alpha1
kind: Package
metadata:
name: grafana
namespace: grafana
spec:
sso:
- name: My Keycloak Client
clientId: demo-client
redirectUris:
- "https://demo.uds.dev/login"
# Customize the name of the generated secret
secretName: my-cool-auth-client
secretTemplate:
# Raw text examples
rawTextClientId: "clientField(clientId)"
rawTextClientSecret: "clientField(secret)"
# JSON example
auth.json: |
{
"client_id": "clientField(clientId)",
"client_secret": "clientField(secret)",
"defaultScopes": clientField(defaultClientScopes).json(),
"redirect_uri": "clientField(redirectUris)[0]",
"bearerOnly": clientField(bearerOnly),
}
# Properties example
auth.properties: |
client-id=clientField(clientId)
client-secret=clientField(secret)
default-scopes=clientField(defaultClientScopes)
redirect-uri=clientField(redirectUris)[0]
# YAML example (uses JSON for the defaultScopes array)
auth.yaml: |
client_id: clientField(clientId)
client_secret: clientField(secret)
default_scopes: clientField(defaultClientScopes).json()
redirect_uri: clientField(redirectUris)[0]
bearer_only: clientField(bearerOnly)
Configuring UDS Core Policy Exemptions
Default policy exemptions are confined to a singular namespace: uds-policy-exemptions
. We find this to be an optimal approach for UDS due to the following reasons:
- Emphasis on Security Impact:
- An exemption has the potential to diminish the overall security stance of the cluster. By isolating these exemptions within a designated namespace, administrators can readily recognize and assess the security implications associated with each exemption.
- Simplified RBAC Maintenance:
- Adopting this pattern streamlines the management of Role-Based Access Control (RBAC) for overseeing exemptions. Placing all UDS exemptions within a dedicated namespace simplifies the task of configuring and maintaining RBAC policies, enhancing overall control and transparency.
- Mitigation of Configuration Risks:
- By restricting exemptions to a specific namespace, the risk of unintentional misconfigurations in RBAC is significantly reduced. This ensures that cluster exemptions are only granted intentionally and within the confines of the designated namespace, minimizing the potential for security vulnerabilities resulting from misconfigured permissions.
Allow All Namespaces
If you find that the default scoping is not the right approach for your cluster, you have the option to configure UDS-CORE
at deploy time to allow exemption CRs in all namespaces:
zarf package deploy zarf-package-uds-core-*.zst --set ALLOW_ALL_NS_EXEMPTIONS=true
You can also achieve this through the uds-config.yaml
:
options:
# options here
shared:
ALLOW_ALL_NS_EXEMPTIONS: "true"
variables:
# package specific variables here
Key Files and Folders
src/pepr/operator/
├── controllers # Core business logic called by the reconciler
│ ├── exemptions # Manages updating Pepr store with exemptions from UDS Exemption
│ ├── istio # Manages Istio VirtualServices and sidecar injection for UDS Packages/Namespace
│ ├── keycloak # Manages Keycloak client syncing
│ └── network # Manages default and generated NetworkPolicies for UDS Packages/Namespace
├── crd
│ ├── generated # Type files generated by `uds run -f src/pepr/tasks.yaml gen-crds`
│ ├── sources # CRD source files
│ ├── migrate.ts # Migrates older versions of UDS Package CRs to new version
│ ├── register.ts # Registers the UDS Package CRD with the Kubernetes API
│ └── validators # Validates Custom Resources with Pepr
├── index.ts # Entrypoint for the UDS Operator
└── reconcilers # Reconciles Custom Resources via the controllers
3 - Configuring Policy Exemptions
By default policy exemptions (UDSExemptions) are only allowed in a single namespace – uds-policy-exemptions
. We recognize this is not a conventional pattern in K8s, but believe it is ideal for UDS for the following reasons:
- highlights the fact that an exemption can reduce the overall security posture of the cluster
- makes maintaining RBAC for controlling exemptions more straightforward
- reduces the risk that an unintentional mis-configuration of RBAC allows a cluster exemption that would otherwise be denied
Allow All Namespaces
If you believe that the default scoping is not the right approach for your cluster, you can configure UDS-CORE at deploy time to allow exemption CRs in all namespaces.
zarf package deploy zarf-package-uds-core-*.zst --set ALLOW_ALL_NS_EXEMPTIONS=true
or via a uds bundle config:
uds-config.yaml
options:
# options here
shared:
ALLOW_ALL_NS_EXEMPTIONS: "true"
variables:
# package specific variables here
4 - User Groups
UDS Core deploys Keycloak which has some preconfigured groups that applications inherit from SSO and IDP configurations.
Applications
Grafana
Grafana maps the groups from Keycloak to it’s internal Admin
and Viewer
groups.
Keycloak Group | Mapped Grafana Group |
---|---|
Admin | Admin |
Auditor | Viewer |
If a user doesn’t belong to either of these Keycloak groups the user will be unauthorized when accessing Grafana.
Neuvector
Neuvector maps the groups from Keycloak to it’s internal admin
and reader
groups.
Keycloak Group | Mapped Neuvector Group |
---|---|
Admin | admin |
Auditor | reader |
Keycloak
Note
All groups are under the Uds Core parent group. Frequently a group will be referred to as Uds Core/Admin or Uds Core/Auditor. In the Keycloak UI this requires an additional click to get down to the sub groups.
Identity Providers ( IDP )
UDS Core ships with a templated Google SAML IDP, more documentation to configure the realmInitEnv
values in uds-identity-config.
Alternatively, the realmInitEnv
can be configured via bundle overrides like in the k3d-standard-bundle.
Configuring your own IDP can be achieved via:
Custom uds-identity-config with a templated realm.json
Keycloak Admin UI and click ops
Custom realm.json for direct import in Keycloak