Kubernetes + prometheus + envoy không thể kết nối

Em chào mọi người ạ, em đang cài prometheus trong kubernetes, đã cài đặt thành công và có thể forward port ra ngoài

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Nhưng khi em dùng envoy proxy để kết nối sang thì nó lại ko kết nối được ạ mặc dù trước đó em đã thử với jaeger và nó chạy ok.

admin:
  address:
    socket_address: {address: 0.0.0.0, port_value: 9901}

static_resources:
  listeners:
  - name: listener1
    address:
      socket_address: {address: 0.0.0.0, port_value: 51051}
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: /app/server-cert.pem
              private_key:
                filename: /app/server-key.pem
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: jaeger_service
              domains: ["jaeger.localhost:30000"]
              routes:
              - match: {prefix: "/"}
                route: {cluster: jaeger, timeout: 60s}
            - name: prometheus_service
              domains: ["prometheus.localhost:30000"]
              routes:
              - match: {prefix: "/"}
                route: {cluster: prometheus, timeout: 60s}

          http_filters:
          - name: envoy.filters.http.header_to_metadata
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.header_to_metadata.v3.Config
              request_rules:
                - cookie: cookie
                  on_header_present:
                    metadata_namespace: envoy.lb
                    key: session_id
                    type: STRING
                  remove: false
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: jaeger
    connect_timeout: 1s
    type: LOGICAL_DNS
    lb_policy: LEAST_REQUEST
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: jaeger
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: jaeger-query.observability.svc.cluster.local
                port_value: 16686
  - name: prometheus
    connect_timeout: 1s
    type: LOGICAL_DNS
    lb_policy: LEAST_REQUEST
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: prometheus
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: prometheus-k8s.monitoring.svc.cluster.local
                port_value: 9090

Đây là các service monitoring:

NAME                                       READY   STATUS             RESTARTS       AGE   IP             NODE             NOMINATED NODE   READINESS GATES
pod/alertmanager-main-0                    2/2     Running            0              47m   10.1.1.40      docker-desktop   <none>           <none>
pod/alertmanager-main-1                    2/2     Running            0              47m   10.1.1.38      docker-desktop   <none>           <none>
pod/alertmanager-main-2                    2/2     Running            0              47m   10.1.1.39      docker-desktop   <none>           <none>
pod/blackbox-exporter-68d54c49dc-kvlrp     3/3     Running            0              47m   10.1.1.32      docker-desktop   <none>           <none>
pod/grafana-6cd5ccdcb8-w8cd8               1/1     Running            0              47m   10.1.1.33      docker-desktop   <none>           <none>
pod/kube-state-metrics-84db6cc79c-wj8n9    3/3     Running            0              47m   10.1.1.34      docker-desktop   <none>           <none>
pod/node-exporter-vvnnx                    1/2     CrashLoopBackOff   14 (63s ago)   47m   192.168.65.4   docker-desktop   <none>           <none>
pod/prometheus-adapter-757f9b4cf9-f8slr    1/1     Running            0              47m   10.1.1.35      docker-desktop   <none>           <none>
pod/prometheus-adapter-757f9b4cf9-t97mf    1/1     Running            0              47m   10.1.1.36      docker-desktop   <none>           <none>
pod/prometheus-k8s-0                       2/2     Running            0              47m   10.1.1.42      docker-desktop   <none>           <none>
pod/prometheus-k8s-1                       2/2     Running            0              47m   10.1.1.41      docker-desktop   <none>           <none>
pod/prometheus-operator-7cf95bc44c-59g75   2/2     Running            0              47m   10.1.1.37      docker-desktop   <none>           <none>
pod/prometheus-service-prometheus-0        2/2     Running            0              47m   10.1.1.44      docker-desktop   <none>           <none>
pod/prometheus-service-prometheus-1        2/2     Running            0              47m   10.1.1.43      docker-desktop   <none>           <none>

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
service/alertmanager-main       ClusterIP   10.98.243.3      <none>        9093/TCP,8080/TCP            47m   app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus
service/alertmanager-operated   ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   47m   app.kubernetes.io/name=alertmanager
service/blackbox-exporter       ClusterIP   10.98.111.173    <none>        9115/TCP,19115/TCP           47m   app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus
service/grafana                 ClusterIP   10.97.218.83     <none>        3000/TCP                     47m   app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus
service/kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP            47m   app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus
service/node-exporter           ClusterIP   None             <none>        9100/TCP                     47m   app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus
service/prometheus-adapter      ClusterIP   10.104.56.110    <none>        443/TCP                      47m   app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus
service/prometheus-k8s          ClusterIP   10.101.238.145   <none>        9090/TCP,8080/TCP            47m   app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus
service/prometheus-operated     ClusterIP   None             <none>        9090/TCP                     47m   app.kubernetes.io/name=prometheus
service/prometheus-operator     ClusterIP   None             <none>        8443/TCP                     47m   app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS                      IMAGES                                                                           SELECTOR
daemonset.apps/node-exporter   1         1         0       1            0           kubernetes.io/os=linux   47m   node-exporter,kube-rbac-proxy   quay.io/prometheus/node-exporter:v1.5.0,quay.io/brancz/kube-rbac-proxy:v0.13.1   app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                                                     IMAGES
                                                                                                                             SELECTOR
deployment.apps/blackbox-exporter     1/1     1            1           47m   blackbox-exporter,module-configmap-reloader,kube-rbac-proxy    quay.io/prometheus/blackbox-exporter:v0.23.0,jimmidyson/configmap-reload:v0.5.0,quay.io/brancz/kube-rbac-proxy:v0.13.1                       app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus
deployment.apps/grafana               1/1     1            1           47m   grafana                                                        grafana/grafana:9.3.1                                                                                                                        app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus
deployment.apps/kube-state-metrics    1/1     1            1           47m   kube-state-metrics,kube-rbac-proxy-main,kube-rbac-proxy-self   registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0,quay.io/brancz/kube-rbac-proxy:v0.13.1,quay.io/brancz/kube-rbac-proxy:v0.13.1   app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus
deployment.apps/prometheus-adapter    2/2     2            2           47m   prometheus-adapter                                             registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.10.0                                                                                app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus
deployment.apps/prometheus-operator   1/1     1            1           47m   prometheus-operator,kube-rbac-proxy                            quay.io/prometheus-operator/prometheus-operator:v0.61.1,quay.io/brancz/kube-rbac-proxy:v0.13.1                                               app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus

NAME                                             DESIRED   CURRENT   READY   AGE   CONTAINERS                                                     IMAGES                                                                                                                                       SELECTOR
replicaset.apps/blackbox-exporter-68d54c49dc     1         1         1       47m   blackbox-exporter,module-configmap-reloader,kube-rbac-proxy    quay.io/prometheus/blackbox-exporter:v0.23.0,jimmidyson/configmap-reload:v0.5.0,quay.io/brancz/kube-rbac-proxy:v0.13.1                       app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus,pod-template-hash=68d54c49dc
replicaset.apps/grafana-6cd5ccdcb8               1         1         1       47m   grafana                                                        grafana/grafana:9.3.1                                                                                                                        app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus,pod-template-hash=6cd5ccdcb8
replicaset.apps/kube-state-metrics-84db6cc79c    1         1         1       47m   kube-state-metrics,kube-rbac-proxy-main,kube-rbac-proxy-self   registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0,quay.io/brancz/kube-rbac-proxy:v0.13.1,quay.io/brancz/kube-rbac-proxy:v0.13.1   app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus,pod-template-hash=84db6cc79c
replicaset.apps/prometheus-adapter-757f9b4cf9    2         2         2       47m   prometheus-adapter                                             registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.10.0                                                                                app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus,pod-template-hash=757f9b4cf9
replicaset.apps/prometheus-operator-7cf95bc44c   1         1         1       47m   prometheus-operator,kube-rbac-proxy                            quay.io/prometheus-operator/prometheus-operator:v0.61.1,quay.io/brancz/kube-rbac-proxy:v0.13.1                                               app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus,pod-template-hash=7cf95bc44c

NAME                                             READY   AGE   CONTAINERS                     IMAGES
statefulset.apps/alertmanager-main               3/3     47m   alertmanager,config-reloader   quay.io/prometheus/alertmanager:v0.24.0,quay.io/prometheus-operator/prometheus-config-reloader:v0.61.1
statefulset.apps/prometheus-k8s                  2/2     47m   prometheus,config-reloader     quay.io/prometheus/prometheus:v2.40.5,quay.io/prometheus-operator/prometheus-config-reloader:v0.61.1
statefulset.apps/prometheus-service-prometheus   2/2     47m   prometheus,config-reloader     quay.io/prometheus/prometheus,quay.io/prometheus-operator/prometheus-config-reloader:v0.61.1

Đây là kết quả trả về khi truy cập url https://prometheus.localhost:30000/

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Nhờ mọi người xem giúp em ạ, em cảm ơn.

             - name: jaeger_service
              domains: ["jaeger.localhost:30000"]
              routes:
              - match: {prefix: "/"}
                route: {cluster: jaeger, timeout: 60s}
            - name: prometheus_service
              domains: ["prometheus.localhost:30000"]
              routes:
              - match: {prefix: "/"}
                route: {cluster: prometheus, timeout: 60s}

Tớ tò mò cái, sao prometheus với jagger lại có chung cổng vậy cậu?
Với cả, cậu thử nhảy vào pod chứa envoy để curl tới prometheus xem :sweat_smile:

Dạ tại vì em để nó khác host á anh, em đặt thêm cái host name vô etc/hosts (hình như vậy). Với em không có cách nào ngoài việc dùng forward port rồi chứ ko dùng envoy nữa. Với lại em cũng không dùng cách này đưa monitoring service vô nữa, hiện em đang dùng istio trong đó có luôn mấy cái monitoring rất dễ dùng ạ :blush:

1 Like
83% thành viên diễn đàn không hỏi bài tập, còn bạn thì sao?