Prometheus와 Grafana 구성

prometheus와 grafana는 helm으로 구성한다.

# add prometheus Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# add grafana Helm repo
helm repo add grafana https://grafana.github.io/helm-charts

Deploy Promethus

kubectl create namespace prometheus

helm install prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"

NAME: prometheus
LAST DEPLOYED: Sat May  8 14:40:52 2021
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.prometheus.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9090


The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.prometheus.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been moved to a global property.  #####
######            use .Values.podSecurityPolicy.enabled with pod-based      #####
######            annotations                                               #####
######            (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.prometheus.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

위에서 Prometheus Server의 endpoint는 다음과 같이 나온다.

The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.prometheus.svc.cluster.local

 

prometheus 설치 상태를 확인한다.

$ kubectl get all -n prometheus

NAME                                                 READY   STATUS    RESTARTS   AGE
pod/prometheus-alertmanager-ccf8f68cd-r2shv          2/2     Running   0          3m45s
pod/prometheus-kube-state-metrics-685b975bb7-d97sh   1/1     Running   0          3m45s
pod/prometheus-node-exporter-82kp4                   1/1     Running   0          3m45s
pod/prometheus-node-exporter-b25n7                   1/1     Running   0          3m45s
pod/prometheus-node-exporter-b2jx7                   1/1     Running   0          3m45s
pod/prometheus-pushgateway-74cb65b858-rlctw          1/1     Running   0          3m45s
pod/prometheus-server-d9fb67455-xztwz                2/2     Running   0          3m45s

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/prometheus-alertmanager         ClusterIP   172.20.144.72    <none>        80/TCP     3m45s
service/prometheus-kube-state-metrics   ClusterIP   172.20.204.212   <none>        8080/TCP   3m45s
service/prometheus-node-exporter        ClusterIP   None             <none>        9100/TCP   3m45s
service/prometheus-pushgateway          ClusterIP   172.20.49.166    <none>        9091/TCP   3m45s
service/prometheus-server               ClusterIP   172.20.58.186    <none>        80/TCP     3m45s

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   3         3         3       3            3           <none>          3m45s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-alertmanager         1/1     1            1           3m45s
deployment.apps/prometheus-kube-state-metrics   1/1     1            1           3m45s
deployment.apps/prometheus-pushgateway          1/1     1            1           3m45s
deployment.apps/prometheus-server               1/1     1            1           3m45s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-alertmanager-ccf8f68cd          1         1         1       3m45s
replicaset.apps/prometheus-kube-state-metrics-685b975bb7   1         1         1       3m45s
replicaset.apps/prometheus-pushgateway-74cb65b858          1         1         1       3m45s
replicaset.apps/prometheus-server-d9fb67455                1         1         1       3m45s

 

grafana를 설치하기 위해서 grafana.yaml을 생성하여 설치한다.

 

mkdir ${HOME}/environment/grafana

cat << EoF > ${HOME}/environment/grafana/grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true
EoF

kubectl create namespace grafana

helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='EKS!sAWSome' \
    --values ${HOME}/environment/grafana/grafana.yaml \
    --set service.type=LoadBalancer

kubectl get all -n grafana

NAME                           READY   STATUS     RESTARTS   AGE
pod/grafana-765fbf745b-c2562   0/1     Init:0/1   0          7s

NAME              TYPE           CLUSTER-IP       EXTERNAL-IP                                                                   PORT(S)        AGE
service/grafana   LoadBalancer   172.20.209.209   a56a19cfa5aa047c28ad99ac6348b60b-647981297.ap-northeast-2.elb.amazonaws.com   80:30360/TCP   7s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   0/1     1            0           7s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-765fbf745b   1         1         0       7s

 

Grafana ELB URL을 확인한다.

export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "http://$ELB"

http://a56a19cfa5aa047c28ad99ac6348b60b-647981297.ap-northeast-2.elb.amazonaws.com

해당 url을 통해 접속한다.