init
This commit is contained in:
Waylon S. Walker
commit
0a4b32dffe
12 changed files with 840 additions and 0 deletions
1
.gitignore
vendored
Normal file
1
.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
.null-ls_806023_main.py
|
||||||
118
README.md
Normal file
118
README.md
Normal file
|
|
@ -0,0 +1,118 @@
|
||||||
|
|
||||||
|
# Monitor Kubernetes Logs with Grafana Alloy and Loki
|
||||||
|
|
||||||
|
> Note this scenario works using the K8s Monitoring Helm chart. This abstracts the need to configure Alloy and deploys best practices for monitoring Kubernetes clusters. The chart supports; metrics, logs, profiling, and tracing. For this scenario, we will use the K8s Monitoring Helm chart to monitor Kubernetes logs.
|
||||||
|
|
||||||
|
This scenario demonstrates how to setup the Kubernetes monitoring helm and Loki. This scenario will install three Helm charts: Loki, Grafana, and k8s-monitoring-helm. Loki will be used to store the logs, Grafana will be used to visualize the logs, and Alloy (k8s-monitoring-helm) will be used to collect three different log sources:
|
||||||
|
* Pod Logs
|
||||||
|
* Kubernetes Events
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
Clone the repository:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/grafana/alloy-scenarios.git
|
||||||
|
```
|
||||||
|
|
||||||
|
Change to the directory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd alloy-scenarios/k8s-logs
|
||||||
|
```
|
||||||
|
|
||||||
|
Next you will need a Kubernetes cluster (In this example, we will configure a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/))
|
||||||
|
|
||||||
|
An example kind cluster configuration is provided in the `kind.yml` file. To create a kind cluster using this configuration, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kind create cluster --config kind.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Lastly you will need to make sure you install Helm on your local machine. You can install Helm by following the instructions [here](https://helm.sh/docs/intro/install/). You will also need to install the Grafana Helm repository:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm repo add grafana https://grafana.github.io/helm-charts
|
||||||
|
```
|
||||||
|
|
||||||
|
## Create the `meta` and `prod` namespaces
|
||||||
|
|
||||||
|
The first step is to create the `meta` and `prod` namespaces. To create the namespaces, run the following commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl create namespace meta && \
|
||||||
|
kubectl create namespace prod
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## Install the Loki Helm Chart
|
||||||
|
|
||||||
|
The first step is to install the Loki Helm chart. This will install Loki in the `meta` namespace. The `loki-values.yml` file contains the configuration for the Loki Helm chart. To install Loki, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm install --values loki-values.yml loki grafana/loki -n meta
|
||||||
|
```
|
||||||
|
|
||||||
|
This installs Loki in monolithic mode. For more information on Loki modes, see the [Loki documentation](https://grafana.com/docs/loki/latest/get-started/deployment-modes/).
|
||||||
|
|
||||||
|
## Install the Grafana Helm Chart
|
||||||
|
|
||||||
|
The next step is to install the Grafana Helm chart. This will install Grafana in the `meta` namespace. The `grafana-values.yml` file contains the configuration for the Grafana Helm chart. To install Grafana, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm install --values grafana-values.yml grafana grafana/grafana --namespace meta
|
||||||
|
```
|
||||||
|
Note that within the `grafana-values.yml` file, the `grafana.ini` configuration is set to use the Loki data source. This is done by setting the `datasources.datasources.yaml` field to the Loki data source configuration.
|
||||||
|
|
||||||
|
## Install the K8s Monitoring Helm Chart
|
||||||
|
|
||||||
|
The final step is to install the K8s monitoring Helm chart. This will install Alloy in the `meta` namespace. The `k8s-monitoring-values.yml` file contains the configuration for the K8s monitoring Helm chart. To install the K8s monitoring Helm chart, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm install --values ./k8s-monitoring-values.yml k8s grafana/k8s-monitoring -n meta --create-namespace
|
||||||
|
```
|
||||||
|
Within the `k8s-monitoring-values.yml` file we declare the Alloy configuration. This configuration specifies the log sources that Alloy will collect logs from. In this scenario, we are collecting logs from two different sources: Pod Logs and Kubernetes Events.
|
||||||
|
|
||||||
|
## Accessing the Grafana UI
|
||||||
|
|
||||||
|
To access the Grafana UI, you will need to port-forward the Grafana pod to your local machine. First, get the name of the Grafana pod:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
```
|
||||||
|
|
||||||
|
Next, port-forward the Grafana pod to your local machine:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl --namespace meta port-forward $POD_NAME 3000
|
||||||
|
```
|
||||||
|
|
||||||
|
Open your browser and go to [http://localhost:3000](http://localhost:3000). You can log in with the default username `admin` and password `adminadminadmin`.
|
||||||
|
|
||||||
|
## Accessing the Alloy UI
|
||||||
|
|
||||||
|
To access the Alloy UI, you will need to port-forward the Alloy pod to your local machine. First, get the name of the Alloy pod:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=alloy-logs,app.kubernetes.io/instance=k8s" -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
```
|
||||||
|
|
||||||
|
Next, port-forward the Alloy pod to your local machine:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl --namespace meta port-forward $POD_NAME 12345
|
||||||
|
```
|
||||||
|
|
||||||
|
## View the logs using Explore Logs in Grafana
|
||||||
|
|
||||||
|
Explore Logs is a new feature in Grafana which provides a queryless way to explore logs. To access Explore Logs. To access Explore logs open a browser and go to [http://localhost:3000/a/grafana-lokiexplore-app](http://localhost:3000/a/grafana-lokiexplore-app).
|
||||||
|
|
||||||
|
## Adding a demo prod app
|
||||||
|
|
||||||
|
The k8s monitoring app is configured to collect logs from two namespaces: `meta` and `prod`. To add a demo prod app, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm install tempo grafana/tempo-distributed -n prod
|
||||||
|
```
|
||||||
|
|
||||||
|
This will install the Tempo distributed tracing system in the `prod` namespace.
|
||||||
166
dashboards.yaml
Normal file
166
dashboards.yaml
Normal file
|
|
@ -0,0 +1,166 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: my-dashboard
|
||||||
|
namespace: meta
|
||||||
|
labels:
|
||||||
|
grafana_dashboard: "1"
|
||||||
|
data:
|
||||||
|
my-dashboard.json: |
|
||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"builtIn": 1,
|
||||||
|
"datasource": {
|
||||||
|
"type": "grafana",
|
||||||
|
"uid": "-- Grafana --"
|
||||||
|
},
|
||||||
|
"enable": true,
|
||||||
|
"hide": true,
|
||||||
|
"iconColor": "rgba(0, 211, 255, 1)",
|
||||||
|
"name": "Annotations & Alerts",
|
||||||
|
"type": "dashboard"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 0,
|
||||||
|
"id": 59,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "tempo",
|
||||||
|
"uid": "cel2jaxx4s4xsf"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"custom": {
|
||||||
|
"align": "auto",
|
||||||
|
"cellOptions": {
|
||||||
|
"type": "auto"
|
||||||
|
},
|
||||||
|
"inspect": false
|
||||||
|
},
|
||||||
|
"mappings": [],
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"color": "green"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "red",
|
||||||
|
"value": 80
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 0,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 1,
|
||||||
|
"options": {
|
||||||
|
"cellHeight": "sm",
|
||||||
|
"footer": {
|
||||||
|
"countRows": false,
|
||||||
|
"fields": "",
|
||||||
|
"reducer": [
|
||||||
|
"sum"
|
||||||
|
],
|
||||||
|
"show": false
|
||||||
|
},
|
||||||
|
"showHeader": true
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.6.1",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "tempo",
|
||||||
|
"uid": "cel2jaxx4s4xsf"
|
||||||
|
},
|
||||||
|
"filters": [
|
||||||
|
{
|
||||||
|
"id": "574a7fa6",
|
||||||
|
"operator": "=",
|
||||||
|
"scope": "span"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"limit": 20,
|
||||||
|
"metricsQueryType": "range",
|
||||||
|
"queryType": "traceqlSearch",
|
||||||
|
"refId": "A",
|
||||||
|
"tableType": "traces"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "traces over last 5 minutes",
|
||||||
|
"type": "table"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "loki",
|
||||||
|
"uid": "P8E80F9AEF21F6940"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 0,
|
||||||
|
"y": 8
|
||||||
|
},
|
||||||
|
"id": 2,
|
||||||
|
"options": {
|
||||||
|
"dedupStrategy": "none",
|
||||||
|
"enableInfiniteScrolling": false,
|
||||||
|
"enableLogDetails": true,
|
||||||
|
"prettifyLogMessage": false,
|
||||||
|
"showCommonLabels": false,
|
||||||
|
"showLabels": false,
|
||||||
|
"showTime": false,
|
||||||
|
"sortOrder": "Descending",
|
||||||
|
"wrapLogMessage": false
|
||||||
|
},
|
||||||
|
"pluginVersion": "11.6.1",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "loki",
|
||||||
|
"uid": "P8E80F9AEF21F6940"
|
||||||
|
},
|
||||||
|
"direction": "backward",
|
||||||
|
"editorMode": "builder",
|
||||||
|
"expr": "{service_name=~\"python-otel-.*\"} |= ``",
|
||||||
|
"queryType": "range",
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "python-otel-logs",
|
||||||
|
"type": "logs"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"preload": false,
|
||||||
|
"schemaVersion": 41,
|
||||||
|
"tags": [],
|
||||||
|
"templating": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"time": {
|
||||||
|
"from": "now-5m",
|
||||||
|
"to": "now"
|
||||||
|
},
|
||||||
|
"timepicker": {},
|
||||||
|
"timezone": "browser",
|
||||||
|
"title": "python-otel-from-configmap",
|
||||||
|
"uid": "fel2uhjhepg5ce",
|
||||||
|
"version": 3
|
||||||
|
}
|
||||||
1
grafana-dashboards
Submodule
1
grafana-dashboards
Submodule
|
|
@ -0,0 +1 @@
|
||||||
|
Subproject commit ad76eac2fbc99c8f6171048abecd8a7faf39297c
|
||||||
43
grafana-values.yml
Normal file
43
grafana-values.yml
Normal file
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
persistence:
|
||||||
|
type: pvc
|
||||||
|
enabled: true
|
||||||
|
|
||||||
|
# DO NOT DO THIS IN PRODUCTION USECASES
|
||||||
|
adminUser: admin
|
||||||
|
adminPassword: adminadminadmin
|
||||||
|
# CONSIDER USING AN EXISTING SECRET
|
||||||
|
# Use an existing secret for the admin user.
|
||||||
|
# admin:
|
||||||
|
## Name of the secret. Can be templated.
|
||||||
|
# existingSecret: ""
|
||||||
|
# userKey: admin-user
|
||||||
|
# passwordKey: admin-password
|
||||||
|
|
||||||
|
service:
|
||||||
|
enabled: true
|
||||||
|
type: ClusterIP
|
||||||
|
|
||||||
|
sidecar:
|
||||||
|
dashboards:
|
||||||
|
enabled: true
|
||||||
|
label: grafana_dashboard
|
||||||
|
labelValue: "1"
|
||||||
|
folder: /tmp/dashboards
|
||||||
|
searchNamespace: ""
|
||||||
|
|
||||||
|
|
||||||
|
datasources:
|
||||||
|
datasources.yaml:
|
||||||
|
apiVersion: 1
|
||||||
|
datasources:
|
||||||
|
- name: Loki
|
||||||
|
type: loki
|
||||||
|
access: proxy
|
||||||
|
orgId: 1
|
||||||
|
url: http://loki-gateway.meta.svc.cluster.local:80
|
||||||
|
basicAuth: false
|
||||||
|
isDefault: false
|
||||||
|
version: 1
|
||||||
|
editable: false
|
||||||
|
|
||||||
25
justfile
Normal file
25
justfile
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
forward_loki:
|
||||||
|
kubectl port-forward --namespace meta svc/loki-gateway 3100:80
|
||||||
|
forward_grafana:
|
||||||
|
#!/bin/bash
|
||||||
|
export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
kubectl --namespace meta port-forward $POD_NAME 3000
|
||||||
|
|
||||||
|
forward_alloy_logs:
|
||||||
|
#!/bin/bash
|
||||||
|
export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=alloy-logs,app.kubernetes.io/instance=k8s" -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
kubectl --namespace meta port-forward $POD_NAME 12345
|
||||||
|
|
||||||
|
forward_otel_collector:
|
||||||
|
#!/bin/bash
|
||||||
|
export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=otel-collector,app.kubernetes.io/instance=k8s" -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
kubectl --namespace meta port-forward $POD_NAME 4317
|
||||||
|
|
||||||
|
get_inotify_max_user_watches:
|
||||||
|
#!/bin/bashcat /proc/sys/fs/inotify/max_user_watches
|
||||||
|
|
||||||
|
export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
# kubectl exec -n <grafana-namespace> <grafana-pod> -- sh -c "ulimit -n && cat /proc/sys/fs/inotify/max_user_watches"
|
||||||
|
kubectl exec -n meta $POD_NAME -- sh -c "ulimit -n && cat /proc/sys/fs/inotify/max_user_watches"
|
||||||
|
|
||||||
|
|
||||||
57
k8s-monitoring-values.yml
Normal file
57
k8s-monitoring-values.yml
Normal file
|
|
@ -0,0 +1,57 @@
|
||||||
|
cluster:
|
||||||
|
name: meta-monitoring-tutorial
|
||||||
|
destinations:
|
||||||
|
- name: loki
|
||||||
|
type: loki
|
||||||
|
url: http://loki-gateway.meta.svc.cluster.local/loki/api/v1/push
|
||||||
|
clusterEvents:
|
||||||
|
enabled: true
|
||||||
|
collector: alloy-logs
|
||||||
|
namespaces: []
|
||||||
|
nodeLogs:
|
||||||
|
enabled: false
|
||||||
|
podLogs:
|
||||||
|
enabled: true
|
||||||
|
gatherMethod: volumes
|
||||||
|
collector: alloy-logs
|
||||||
|
labelsToKeep: ["app_kubernetes_io_name", "container", "instance", "job", "level", "namespace", "service_name", "service_namespace", "deployment_environment", "deployment_environment_name", 'span_id', 'trace_id']
|
||||||
|
operators:
|
||||||
|
- type: json_parser
|
||||||
|
id: parse_json
|
||||||
|
parse_from: body
|
||||||
|
- type: add_labels
|
||||||
|
labels:
|
||||||
|
trace_id: ${body.trace_id}
|
||||||
|
span_id: ${body.span_id}
|
||||||
|
service_name: ${body.service_name}
|
||||||
|
structuredMetadata:
|
||||||
|
pod: pod # Set structured metadata "pod" from label "pod"
|
||||||
|
namespaces: []
|
||||||
|
# Collectors
|
||||||
|
alloy-singleton:
|
||||||
|
enabled: false
|
||||||
|
alloy-metrics:
|
||||||
|
enabled: false
|
||||||
|
nodeExporter:
|
||||||
|
enabled: true
|
||||||
|
kube-state-metrics:
|
||||||
|
enabled: true
|
||||||
|
nodeLogs:
|
||||||
|
enabled: true
|
||||||
|
kubeletMetrics:
|
||||||
|
enabled: true
|
||||||
|
cAdvisor:
|
||||||
|
enabled: true
|
||||||
|
dashboards:
|
||||||
|
enabled: true
|
||||||
|
alloy-logs:
|
||||||
|
enabled: true
|
||||||
|
alloy:
|
||||||
|
mounts:
|
||||||
|
varlog: true
|
||||||
|
clustering:
|
||||||
|
enabled: false
|
||||||
|
alloy-profiles:
|
||||||
|
enabled: false
|
||||||
|
alloy-receiver:
|
||||||
|
enabled: false
|
||||||
78
loki-values.yml
Normal file
78
loki-values.yml
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
---
|
||||||
|
loki:
|
||||||
|
auth_enabled: false
|
||||||
|
commonConfig:
|
||||||
|
replication_factor: 1
|
||||||
|
schemaConfig:
|
||||||
|
configs:
|
||||||
|
- from: 2024-04-01
|
||||||
|
store: tsdb
|
||||||
|
object_store: s3
|
||||||
|
schema: v13
|
||||||
|
index:
|
||||||
|
prefix: loki_index_
|
||||||
|
period: 24h
|
||||||
|
ingester:
|
||||||
|
chunk_encoding: snappy
|
||||||
|
tracing:
|
||||||
|
enabled: true
|
||||||
|
pattern_ingester:
|
||||||
|
enabled: true
|
||||||
|
limits_config:
|
||||||
|
allow_structured_metadata: true
|
||||||
|
volume_enabled: true
|
||||||
|
ruler:
|
||||||
|
enable_api: true
|
||||||
|
querier:
|
||||||
|
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
|
||||||
|
max_concurrent: 4
|
||||||
|
|
||||||
|
minio:
|
||||||
|
enabled: true
|
||||||
|
|
||||||
|
deploymentMode: SingleBinary
|
||||||
|
singleBinary:
|
||||||
|
replicas: 1
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpu: 4
|
||||||
|
memory: 4Gi
|
||||||
|
requests:
|
||||||
|
cpu: 2
|
||||||
|
memory: 2Gi
|
||||||
|
extraEnv:
|
||||||
|
# Keep a little bit lower than memory limits
|
||||||
|
- name: GOMEMLIMIT
|
||||||
|
value: 3750MiB
|
||||||
|
|
||||||
|
chunksCache:
|
||||||
|
# default is 500MB, with limited memory keep this smaller
|
||||||
|
writebackSizeLimit: 10MB
|
||||||
|
|
||||||
|
|
||||||
|
# Zero out replica counts of other deployment modes
|
||||||
|
backend:
|
||||||
|
replicas: 0
|
||||||
|
read:
|
||||||
|
replicas: 0
|
||||||
|
write:
|
||||||
|
replicas: 0
|
||||||
|
|
||||||
|
ingester:
|
||||||
|
replicas: 0
|
||||||
|
querier:
|
||||||
|
replicas: 0
|
||||||
|
queryFrontend:
|
||||||
|
replicas: 0
|
||||||
|
queryScheduler:
|
||||||
|
replicas: 0
|
||||||
|
distributor:
|
||||||
|
replicas: 0
|
||||||
|
compactor:
|
||||||
|
replicas: 0
|
||||||
|
indexGateway:
|
||||||
|
replicas: 0
|
||||||
|
bloomCompactor:
|
||||||
|
replicas: 0
|
||||||
|
bloomGateway:
|
||||||
|
replicas: 0
|
||||||
100
main.py
Normal file
100
main.py
Normal file
|
|
@ -0,0 +1,100 @@
|
||||||
|
import time
|
||||||
|
import random
|
||||||
|
from flask import Flask
|
||||||
|
from opentelemetry import trace
|
||||||
|
from opentelemetry.sdk.trace import TracerProvider
|
||||||
|
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||||
|
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||||
|
from opentelemetry.instrumentation.flask import FlaskInstrumentor
|
||||||
|
from opentelemetry.sdk.resources import Resource
|
||||||
|
import logging
|
||||||
|
from opentelemetry.trace import get_current_span
|
||||||
|
|
||||||
|
|
||||||
|
# Configure tracing
|
||||||
|
trace.set_tracer_provider(
|
||||||
|
TracerProvider(resource=Resource.create({"service.name": "my-flask-app"}))
|
||||||
|
)
|
||||||
|
tracer = trace.get_tracer(__name__)
|
||||||
|
span_processor = BatchSpanProcessor(OTLPSpanExporter())
|
||||||
|
trace.get_tracer_provider().add_span_processor(span_processor)
|
||||||
|
|
||||||
|
|
||||||
|
# Setup logging with trace_id
|
||||||
|
class TraceIdFilter(logging.Filter):
|
||||||
|
def filter(self, record):
|
||||||
|
span = trace.get_current_span()
|
||||||
|
ctx = span.get_span_context()
|
||||||
|
if ctx.trace_id != 0:
|
||||||
|
record.trace_id = format(ctx.trace_id, "032x")
|
||||||
|
record.span_id = format(ctx.span_id, "016x")
|
||||||
|
else:
|
||||||
|
record.trace_id = None
|
||||||
|
record.span_id = None
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
formatter = logging.Formatter(
|
||||||
|
'{"message": "%(message)s", "trace_id": "%(trace_id)s", "span_id": "%(span_id)s"}'
|
||||||
|
)
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(formatter)
|
||||||
|
handler.addFilter(TraceIdFilter())
|
||||||
|
|
||||||
|
logger = logging.getLogger("myapp")
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
logger.addHandler(handler)
|
||||||
|
|
||||||
|
# Setup Tracer
|
||||||
|
resource = Resource(
|
||||||
|
attributes={
|
||||||
|
"service.name": "python-otel-demo-app", # 👈 choose any name you like
|
||||||
|
}
|
||||||
|
)
|
||||||
|
trace.set_tracer_provider(TracerProvider(resource=resource))
|
||||||
|
tracer = trace.get_tracer(__name__)
|
||||||
|
|
||||||
|
# Configure OTLP HTTP exporter
|
||||||
|
otlp_exporter = OTLPSpanExporter(
|
||||||
|
endpoint="http://otel-collector.meta.svc:4318/v1/traces",
|
||||||
|
# insecure=True,
|
||||||
|
)
|
||||||
|
span_processor = BatchSpanProcessor(otlp_exporter)
|
||||||
|
trace.get_tracer_provider().add_span_processor(span_processor)
|
||||||
|
|
||||||
|
# Flask App
|
||||||
|
app = Flask(__name__)
|
||||||
|
FlaskInstrumentor().instrument_app(app)
|
||||||
|
|
||||||
|
|
||||||
|
def get_trace_id():
|
||||||
|
span = get_current_span()
|
||||||
|
if span:
|
||||||
|
return span.get_span_context().trace_id
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/")
|
||||||
|
def root():
|
||||||
|
with tracer.start_as_current_span("handle_homepage"):
|
||||||
|
logger.info("handling request", extra={"trace_id": get_trace_id()})
|
||||||
|
time.sleep(random.random() * 0.05) # simulate request parsing
|
||||||
|
fetch_user()
|
||||||
|
calculate_recommendations()
|
||||||
|
return "Hello from / with traces!"
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_user():
|
||||||
|
with tracer.start_as_current_span("fetch_user"):
|
||||||
|
time.sleep(random.random() * 0.1) # simulate DB query
|
||||||
|
|
||||||
|
|
||||||
|
def calculate_recommendations():
|
||||||
|
with tracer.start_as_current_span("calculate_recommendations"):
|
||||||
|
for i in range(2):
|
||||||
|
with tracer.start_as_current_span(f"score_item_{i}"):
|
||||||
|
time.sleep(random.random())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
app.run(host="0.0.0.0", port=5000)
|
||||||
84
otel-collector.yaml
Normal file
84
otel-collector.yaml
Normal file
|
|
@ -0,0 +1,84 @@
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: otel-collector
|
||||||
|
namespace: meta
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: otel-collector
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: otel-collector
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: otel-collector
|
||||||
|
image: otel/opentelemetry-collector-contrib:0.81.0
|
||||||
|
args: ["--config=/etc/otel/otel-collector-config.yaml"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: otel-config
|
||||||
|
mountPath: /etc/otel
|
||||||
|
volumes:
|
||||||
|
- name: otel-config
|
||||||
|
configMap:
|
||||||
|
name: otel-collector-config
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: otel-collector-config
|
||||||
|
namespace: meta
|
||||||
|
data:
|
||||||
|
otel-collector-config.yaml: |
|
||||||
|
receivers:
|
||||||
|
otlp:
|
||||||
|
protocols:
|
||||||
|
grpc:
|
||||||
|
endpoint: 0.0.0.0:4317
|
||||||
|
http:
|
||||||
|
endpoint: 0.0.0.0:4318
|
||||||
|
|
||||||
|
processors:
|
||||||
|
spanmetrics:
|
||||||
|
dimensions:
|
||||||
|
- name: operation
|
||||||
|
default: unknown
|
||||||
|
metrics_exporter: prometheus
|
||||||
|
aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"
|
||||||
|
|
||||||
|
exporters:
|
||||||
|
otlp:
|
||||||
|
endpoint: tempo:4317
|
||||||
|
tls:
|
||||||
|
insecure: true
|
||||||
|
prometheus:
|
||||||
|
endpoint: "0.0.0.0:8889"
|
||||||
|
|
||||||
|
service:
|
||||||
|
pipelines:
|
||||||
|
traces:
|
||||||
|
receivers: [otlp]
|
||||||
|
processors: [spanmetrics]
|
||||||
|
exporters: [otlp]
|
||||||
|
metrics:
|
||||||
|
receivers: []
|
||||||
|
exporters: [prometheus]
|
||||||
|
processors: []
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: otel-collector
|
||||||
|
namespace: meta
|
||||||
|
spec:
|
||||||
|
ports:
|
||||||
|
- name: grpc
|
||||||
|
port: 4317
|
||||||
|
targetPort: 4317
|
||||||
|
- name: http
|
||||||
|
port: 4318
|
||||||
|
targetPort: 4318
|
||||||
|
selector:
|
||||||
|
app: otel-collector
|
||||||
2
prometheus-values.yaml
Normal file
2
prometheus-values.yaml
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
grafana:
|
||||||
|
enabled: true
|
||||||
165
pyservice.yaml
Normal file
165
pyservice.yaml
Normal file
|
|
@ -0,0 +1,165 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: my-python-app
|
||||||
|
namespace: meta
|
||||||
|
data:
|
||||||
|
main.py: |
|
||||||
|
import time
|
||||||
|
import random
|
||||||
|
from flask import Flask
|
||||||
|
from opentelemetry import trace
|
||||||
|
from opentelemetry.sdk.trace import TracerProvider
|
||||||
|
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||||
|
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||||
|
from opentelemetry.instrumentation.flask import FlaskInstrumentor
|
||||||
|
from opentelemetry.sdk.resources import Resource
|
||||||
|
import logging
|
||||||
|
from opentelemetry.trace import get_current_span
|
||||||
|
SERVICE_NAME = "my-flask-app" # match your OTEL resource name
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
trace.set_tracer_provider(
|
||||||
|
TracerProvider(resource=Resource.create({"service.name": "my-flask-app"}))
|
||||||
|
)
|
||||||
|
tracer = trace.get_tracer(__name__)
|
||||||
|
span_processor = BatchSpanProcessor(OTLPSpanExporter())
|
||||||
|
trace.get_tracer_provider().add_span_processor(span_processor)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
class TraceIdFilter(logging.Filter):
|
||||||
|
def filter(self, record):
|
||||||
|
span = trace.get_current_span()
|
||||||
|
ctx = span.get_span_context()
|
||||||
|
if ctx.trace_id != 0:
|
||||||
|
record.trace_id = format(ctx.trace_id, "032x")
|
||||||
|
record.span_id = format(ctx.span_id, "016x")
|
||||||
|
record.service_name = SERVICE_NAME
|
||||||
|
else:
|
||||||
|
record.trace_id = None
|
||||||
|
record.span_id = None
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
formatter = logging.Formatter(
|
||||||
|
'{"message": "%(message)s", "trace_id": "%(trace_id)s", "span_id": "%(span_id)s", "service_name": "my-flask-app"}'
|
||||||
|
)
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(formatter)
|
||||||
|
handler.addFilter(TraceIdFilter())
|
||||||
|
|
||||||
|
logger = logging.getLogger("myapp")
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
logger.addHandler(handler)
|
||||||
|
|
||||||
|
|
||||||
|
resource = Resource(
|
||||||
|
attributes={
|
||||||
|
"service.name": "python-otel-demo-app", # 👈 choose any name you like
|
||||||
|
}
|
||||||
|
)
|
||||||
|
trace.set_tracer_provider(TracerProvider(resource=resource))
|
||||||
|
tracer = trace.get_tracer(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
otlp_exporter = OTLPSpanExporter(
|
||||||
|
endpoint="http://otel-collector.meta.svc:4318/v1/traces",
|
||||||
|
# insecure=True,
|
||||||
|
)
|
||||||
|
span_processor = BatchSpanProcessor(otlp_exporter)
|
||||||
|
trace.get_tracer_provider().add_span_processor(span_processor)
|
||||||
|
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
FlaskInstrumentor().instrument_app(app)
|
||||||
|
|
||||||
|
|
||||||
|
def get_trace_id():
|
||||||
|
span = get_current_span()
|
||||||
|
if span:
|
||||||
|
return span.get_span_context().trace_id
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/")
|
||||||
|
def root():
|
||||||
|
with tracer.start_as_current_span("handle_homepage"):
|
||||||
|
logger.info("handling request", extra={"trace_id": get_trace_id()})
|
||||||
|
time.sleep(random.random() * 0.05) # simulate request parsing
|
||||||
|
fetch_user()
|
||||||
|
calculate_recommendations()
|
||||||
|
return "Hello from / with traces!"
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_user():
|
||||||
|
with tracer.start_as_current_span("fetch_user"):
|
||||||
|
time.sleep(random.random() * 0.1) # simulate DB query
|
||||||
|
|
||||||
|
|
||||||
|
def calculate_recommendations():
|
||||||
|
with tracer.start_as_current_span("calculate_recommendations"):
|
||||||
|
for i in range(2):
|
||||||
|
with tracer.start_as_current_span(f"score_item_{i}"):
|
||||||
|
time.sleep(random.random())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
app.run(host="0.0.0.0", port=5000)
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: python-otel
|
||||||
|
namespace: meta
|
||||||
|
labels:
|
||||||
|
app: python-otel
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: python-otel
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: python-otel
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: python
|
||||||
|
image: python
|
||||||
|
command: ["/bin/sh", "-c"]
|
||||||
|
args:
|
||||||
|
- |
|
||||||
|
python -m venv .venv && \
|
||||||
|
.venv/bin/python -m pip install --upgrade pip && \
|
||||||
|
.venv/bin/python -m pip install \
|
||||||
|
opentelemetry-sdk \
|
||||||
|
opentelemetry-exporter-otlp \
|
||||||
|
opentelemetry-instrumentation-flask \
|
||||||
|
flask && \
|
||||||
|
.venv/bin/python /app/main.py
|
||||||
|
volumeMounts:
|
||||||
|
- name: app-code
|
||||||
|
mountPath: /app
|
||||||
|
env:
|
||||||
|
- name: OTEL_EXPORTER_OTLP_ENDPOINT
|
||||||
|
value: http://otel-collector.meta.svc:4318
|
||||||
|
- name: OTEL_EXPORTER_OTLP_INSECURE
|
||||||
|
value: "true"
|
||||||
|
volumes:
|
||||||
|
- name: app-code
|
||||||
|
configMap:
|
||||||
|
name: my-python-app
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: python-otel
|
||||||
|
namespace: meta
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
app: python-otel
|
||||||
|
ports:
|
||||||
|
- port: 80
|
||||||
|
targetPort: 5000
|
||||||
Loading…
Add table
Add a link
Reference in a new issue