fix(grafana): fixed false-positive NoData state

btw, they are incorrect

README.md

@@ -32,7 +32,7 @@ Table Report:
 
 #### Warning
 
-Please note that containers will use ports from 13241 to 13245 and 8080, so there is must be no listeners on this ports range.
+Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
 
 #### Configure
 
@@ -66,7 +66,7 @@ docker compose up -d --build
 
 #### Structure
 
-- **backend**: [127.0.0.1:8080](http://127.0.0.1:8080) -> `8080`
+- **backend**: [127.0.0.1:13240](http://127.0.0.1:13240) -> `8080`
   - Depends on: `postgres`, `redis`, `minio`, `backend-initdb`
 - **backend-initdb**
   - Depends on: `postgres`, `redis`, `minio`
@@ -139,7 +139,7 @@ You may say: "For what we need a lot of complex technologies for now". I have an
 
 ### Restful API
 
-API Base endpoint when deployed with default docker compose: [127.0.0.1:8080](http://127.0.0.1:8080), also see [docs](#openapi-docs).
+API Base endpoint when deployed with default docker compose: [127.0.0.1:13240](http://127.0.0.1:13240), also see [docs](#openapi-docs).
 
 ### Admin panel
 
@@ -152,9 +152,13 @@ Link: [t.me/adnova_bot](https://t.me/adnova_bot)
 Basic commands:
 
 `/start` - Start the bot and authenticate as advertiser
+
 `/help` - Get list of all commands
+
 `/campaigns` - Manage advertiser campaigns (only after authentication)
+
 `/statistics` - See advertiser overall statistics (only after authentication)
+
 `/logout` - Logout of current advertiser account (only after authentication)
 
 See [this](#telegram-bot-1).
@@ -234,9 +238,9 @@ Moderation implemented via report system. Client goes to `/report` ([see OpenAPI
 Also admin user (whose credentials specified lower) can add new staff members and even create a specified group for them (this is built-in django capabilities).
 Report has four states: Sent, Under review, Took action and Skipped. Admin panel has filtration by states and by flagged by llm status.
 
-Admin panel when deployed with docker compose (by default): [localhost:8080/admin/](http://localhost:8080/admin/)
+Admin panel when deployed with docker compose (by default): [localhost:13240/admin/](http://localhost:13240/admin/)
 
-Reports list when deployed with docker compose (requires authentication): [localhost:8080/admin/campaign/campaignreport/](http://localhost:8080/admin/campaign/campaignreport/)
+Reports list when deployed with docker compose (requires authentication): [localhost:13240/admin/campaign/campaignreport/](http://localhost:13240/admin/campaign/campaignreport/)
 
 Default username: `admin`
 
@@ -276,11 +280,11 @@ Demonstration:
 
 ### OpenAPI docs
 
-When deployed with default docker compose: [localhost:8080/docs](http://localhost:8080/docs)
+When deployed with default docker compose: [localhost:13240/docs](http://localhost:13240/docs)
 
 ### Healthcheck endpoint
 
-When deployed with default docker compose: [localhost:8080/health](http://localhost:8080/health)
+When deployed with default docker compose: [localhost:13240/health](http://localhost:13240/health)
 
 Lets developers easily understand and identify problem and users check services health.
 

compose.yaml

@@ -5,6 +5,9 @@ services:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     depends_on:
       backend-initdb:
         restart: false
@@ -18,10 +21,6 @@ services:
         restart: false
         condition: service_healthy
         required: true
-      minio:
-        restart: false
-        condition: service_healthy
-        required: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
@@ -30,16 +29,20 @@ services:
     ports:
       - name: web
         target: 8080
-        published: 8080
+        published: 13240
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
     restart: unless-stopped
+    shm_size: 4mb
 
   backend-initdb:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     command: ./scripts/initdb
     depends_on:
       postgres:
@@ -50,27 +53,27 @@ services:
         restart: false
         condition: service_healthy
         required: true
-      minio:
-        restart: false
-        condition: service_healthy
-        required: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
       - path: ./infrastructure/backend/.env
         required: false
+    shm_size: 4mb
 
   backend-staticfiles:
     build:
       context: ./services/backend
       dockerfile: Dockerfile.staticfiles
+      tags:
+        - adnova-backend-staticfiles:latest
+      pull: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
       - path: ./infrastructure/backend/.env
         required: false
     healthcheck:
-      test: ["CMD", "service", "nginx", "status", "||", " exit 1"]
+      test: ["CMD-SHELL", "nginx", "-t", "||", "exit 1"]
       interval: 1m30s
       timeout: 5s
       start_period: 5s
@@ -84,11 +87,15 @@ services:
         protocol: tcp
         app_protocol: http
     restart: unless-stopped
+    shm_size: 4mb
 
   backend-celery-worker:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     command: celery -A config worker -l INFO
     depends_on:
       redis:
@@ -108,6 +115,7 @@ services:
       start_period: 10s
       start_interval: 2s
     restart: unless-stopped
+    shm_size: 4mb
 
   celery-exporter:
     image: docker.io/danihodovic/celery-exporter:0.12.2
@@ -122,7 +130,10 @@ services:
         required: true
       - path: ./infrastructure/celery-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
+    shm_size: 4mb
 
   telegram_bot:
     build:
@@ -130,6 +141,7 @@ services:
       dockerfile: Dockerfile
       tags:
         - adnova-telegram_bot:latest
+      pull: true
     depends_on:
       backend:
         restart: false
@@ -148,7 +160,10 @@ services:
         required: true
       - path: ./infrastructure/telegram_bot/.env
         required: false
+    profiles:
+      - telegram_bot
     restart: unless-stopped
+    shm_size: 4mb
 
   redis:
     image: docker.io/redis:7-alpine3.21
@@ -174,6 +189,7 @@ services:
       - type: volume
         source: redis_data
         target: /data
+        read_only: false
 
   redis-exporter:
     image: docker.io/oliver006/redis_exporter:v1.74.0-alpine
@@ -187,6 +203,8 @@ services:
         required: true
       - path: ./infrastructure/redis-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
 
@@ -201,7 +219,7 @@ services:
       - path: ./infrastructure/postgres/.env
         required: false
     healthcheck:
-      test: ["CMD", "pg_isready"]
+      test: ["CMD", "pg_isready", "-U", "postgres"]
       interval: 1m30s
       timeout: 5s
       start_period: 5s
@@ -209,14 +227,12 @@ services:
       retries: 5
     oom_kill_disable: true
     restart: unless-stopped
-    secrets:
-      - source: postgres_password
-        target: /run/secrets/postgres_password
     shm_size: 128mb
     volumes:
       - type: volume
         source: postgres_data
         target: /var/lib/postgresql/data
+        read_only: false
 
   postgres-exporter:
     image: quay.io/prometheuscommunity/postgres-exporter:v0.17.1
@@ -230,11 +246,13 @@ services:
         required: true
       - path: ./infrastructure/postgres-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
 
   pgadmin:
-    image: docker.io/dpage/pgadmin4:9.5
+    image: docker.io/dpage/pgadmin4:9.6
     configs:
       - source: pgadmin_servers_config
         target: /pgadmin4/servers.json
@@ -262,22 +280,22 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
-    secrets:
-      - source: pgadmin_password
-        target: /run/secrets/pgadmin_password
     shm_size: 4mb
     volumes:
       - type: volume
         source: pgadmin_data
         target: /var/lib/pgadmin
+        read_only: false
 
   grafana:
     image: docker.io/grafana/grafana-oss:12.0.2
+    entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
     configs:
       - source: grafana_config
         target: /usr/share/grafana/conf/defaults.ini
-    entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
     healthcheck:
       test: ["CMD", "wget", "-O", "-", "http://localhost:3000/api/health"]
       interval: 1m30s
@@ -292,18 +310,23 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
     volumes:
       - type: volume
         source: grafana_data
         target: /var/lib/grafana
+        read_only: false
       - type: bind
         source: ./infrastructure/grafana/provisioning
         target: /etc/grafana/provisioning
+        read_only: true
       - type: bind
         source: ./infrastructure/grafana/scripts
         target: /etc/grafana/scripts
+        read_only: true
 
   prometheus:
     image: docker.io/prom/prometheus:v3.5.0
@@ -325,12 +348,15 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
     volumes:
       - type: volume
         source: prometheus_data
         target: /prometheus
+        read_only: false
 
   minio:
     image: docker.io/minio/minio:RELEASE.2025-07-18T21-56-31Z
@@ -360,11 +386,15 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - minio
     restart: unless-stopped
+    shm_size: 4mb
     volumes:
       - type: volume
         source: minio_data
         target: /data
+        read_only: false
 
   zipkin:
     image: docker.io/openzipkin/zipkin:3
@@ -382,18 +412,17 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
-      - name: api
+    profiles:
-        target: 9411
+      - observability
-        published: 13247
-        host_ip: 127.0.0.1
-        protocol: tcp
-        app_protocol: http
     restart: unless-stopped
+    shm_size: 4mb
 
   loadtest:
     build:
       context: ./services/loadtest
       dockerfile: Dockerfile
+      tags:
+        - adnova-loadtest:latest
     depends_on:
       backend:
         restart: false
@@ -411,7 +440,52 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - loadtest
     restart: unless-stopped
+    shm_size: 4mb
+
+  proxy:
+    image: docker.io/caddy:2-alpine
+    configs:
+      - source: caddy_config
+        target: /etc/caddy/Caddyfile
+    ports:
+      - name: http
+        target: 80
+        published: 80
+        host_ip: 0.0.0.0
+        protocol: tcp
+        app_protocol: http
+      - name: https
+        target: 443
+        published: 443
+        host_ip: 0.0.0.0
+        protocol: tcp
+        app_protocol: http
+      - name: http3
+        target: 443
+        published: 443
+        host_ip: 0.0.0.0
+        protocol: udp
+        app_protocol: http
+    profiles:
+      - proxy
+    restart: unless-stopped
+    shm_size: 4mb
+    volumes:
+      - type: volume
+        source: caddy_data
+        target: /data
+        read_only: false
+      - type: volume
+        source: caddy_config
+        target: /config
+        read_only: false
+      - type: bind
+        source: ./infrastructure/caddy/static
+        target: /var/www
+        read_only: true
 
 volumes:
   redis_data:
@@ -420,6 +494,8 @@ volumes:
   grafana_data:
   prometheus_data:
   minio_data:
+  caddy_data:
+  caddy_config:
 
 configs:
   redis_config:
@@ -432,9 +508,5 @@ configs:
     file: ./infrastructure/grafana/grafana.ini
   prometheus_config:
     file: ./infrastructure/prometheus/prometheus.yaml
-
+  caddy_config:
-secrets:
+    file: ./infrastructure/caddy/Caddyfile
-  postgres_password:
-    file: ./infrastructure/postgres/password
-  pgadmin_password:
-    file: ./infrastructure/pgadmin/password

infrastructure/caddy/Caddyfile

@@ -0,0 +1,68 @@
+{
+	admin :2019
+
+	metrics {
+		per_host
+	}
+}
+
+(basic-auth) {
+	basic_auth {
+		admin $2a$14$2zQilpLka2h8Sn1mmOLAAezwDN8Zy8Ta36WECk4qt5MTn3CWksR0m
+	}
+}
+
+adnova.itqdev.xyz {
+	@healthPath path /health /health/*
+	handle @healthPath {
+		import basic-auth
+		reverse_proxy http://backend:8080
+	}
+
+	handle_path /static/* {
+		reverse_proxy http://backend-staticfiles:80
+	}
+
+	reverse_proxy http://backend:8080
+}
+
+:8080 {
+	reverse_proxy http://backend:8080
+}
+
+admin.adnova.itqdev.xyz {
+	import basic-auth
+
+	root * /var/www/admin
+	file_server
+}
+
+loadtest.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://loadtest:5001
+}
+
+grafana.adnova.itqdev.xyz {
+	reverse_proxy http://grafana:3000
+}
+
+pgadmin.adnova.itqdev.xyz {
+	reverse_proxy http://pgadmin:80
+}
+
+zipkin.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://zipkin:9411
+}
+
+prometheus.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://prometheus:9090
+}
+
+minio.adnova.itqdev.xyz {
+	reverse_proxy http://minio:9000
+}

infrastructure/caddy/static/admin/index.html

@@ -0,0 +1,209 @@
+<!DOCTYPE html>
+<html lang="en" class="dark">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>AdNova Admin Resources</title>
+    <script src="https://cdn.tailwindcss.com"></script>
+    <style>
+        body {
+            font-family: 'Inter', sans-serif;
+        }
+
+        @import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');
+    </style>
+</head>
+
+<body class="bg-slate-900 text-gray-300">
+    <div class="container mx-auto p-4 md:p-8">
+        <header class="mb-8">
+            <h1 class="text-4xl font-bold text-white tracking-tight">AdNova Admin Resources Dashboard</h1>
+            <p class="text-indigo-400">Quick access to essential tools and credentials.</p>
+        </header>
+
+        <div id="resources-container" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
+        </div>
+    </div>
+
+    <script>
+        const resources = [
+            {
+                category: "Development Tools",
+                items: [
+                    {
+                        name: "Gitea Repository",
+                        link: "https://git.itqdev.xyz/PROD.2025/AdNova",
+                        description: "Main monorepo on Gitea."
+                    },
+                    {
+                        name: "Rest API Docs",
+                        link: "https://adnova.itqdev.xyz/docs",
+                        description: "Rest API Docs on Swagger."
+                    },
+                    {
+                        name: "Django Admin",
+                        link: "https://adnova.itqdev.xyz/admin/",
+                        username: "admin",
+                        password: "admin",
+                        description: "Django Admin panel."
+                    },
+                    {
+                        name: "Loadtest",
+                        link: "https://loadtest.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Stress test utility with multiple profiles and ability to load mocks."
+                    }
+                ]
+            },
+            {
+                category: "Monitoring",
+                items: [
+                    {
+                        name: "Grafana",
+                        link: "https://grafana.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "proooooood",
+                        description: "Real-time system metrics and dashboards."
+                    },
+                    {
+                        name: "Prometheus",
+                        link: "https://prometheus.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Real-time system metrics and scrape targets."
+                    },
+                    {
+                        name: "Zipkin",
+                        link: "https://zipkin.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Latest traces from production."
+                    },
+                    {
+                        name: "Django Silk",
+                        link: "https://adnova.itqdev.xyz/silk/",
+                        username: "admin",
+                        password: "admin",
+                        description: "App profiles and performance info in production, need to authenticate first as staff in Django Admin."
+                    },
+                    {
+                        name: "Adnova Alerts",
+                        link: "https://t.me/adnova_alerts",
+                        description: "Telegram channel where alerts are posted."
+                    },
+                    {
+                        name: "Healthcheck",
+                        link: "https://adnova.itqdev.xyz/health",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Summary of integration status between services."
+                    },
+                    {
+                        name: "Uptime-Kuma Dashboard",
+                        link: "https://status.adnova.itqdev.xyz",
+                        description: "External monitoring of AdNova."
+                    }
+                ]
+            },
+            {
+                category: "Databases",
+                items: [
+                    {
+                        name: "Pgadmin",
+                        link: "https://pgadmin.adnova.itqdev.xyz",
+                        username: "admin@mail.com",
+                        password: "password",
+                        description: "Access to production database."
+                    }
+                ]
+            }
+        ]
+
+        function copyToClipboard ( text )
+        {
+            const textarea = document.createElement( 'textarea' )
+            textarea.value = text
+            document.body.appendChild( textarea )
+            textarea.select()
+            document.execCommand( 'copy' )
+            document.body.removeChild( textarea )
+            alert( 'Copied to clipboard!' )
+        }
+
+        function renderResources ()
+        {
+            const container = document.getElementById( 'resources-container' )
+            container.innerHTML = ''
+
+            resources.forEach( categoryData =>
+            {
+                const categorySection = document.createElement( 'div' )
+                categorySection.className = 'col-span-full mb-4'
+                categorySection.innerHTML = `
+                    <h2 class="text-2xl font-semibold text-white mb-4 border-b border-slate-700 pb-2">
+                        ${ categoryData.category }
+                    </h2>
+                `
+                container.appendChild( categorySection )
+
+                categoryData.items.forEach( resource =>
+                {
+                    const resourceCard = document.createElement( 'div' )
+                    resourceCard.className = 'bg-slate-800 rounded-lg p-6 shadow-lg flex flex-col justify-between'
+
+                    let credentialsHtml = ''
+                    if ( resource.username || resource.password )
+                    {
+                        credentialsHtml = `
+                            <div class="mt-4 pt-4 border-t border-slate-700">
+                                <h4 class="text-sm font-medium text-slate-400 mb-2">Credentials:</h4>
+                                ${ resource.username ? `
+                                    <div class="flex items-center text-sm mb-2">
+                                        <span class="font-semibold text-slate-300 w-24">Username:</span>
+                                        <span id="user-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.username }</span>
+                                        <button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.username }">Copy</button>
+                                    </div>` : '' }
+                                ${ resource.password ? `
+                                    <div class="flex items-center text-sm">
+                                        <span class="font-semibold text-slate-300 w-24">Password:</span>
+                                        <span id="pass-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.password }</span>
+                                        <button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.password }">Copy</button>
+                                    </div>` : '' }
+                            </div>
+                        `
+                    }
+
+                    resourceCard.innerHTML = `
+                        <div>
+                            <h3 class="text-xl font-semibold text-white mb-2">${ resource.name }</h3>
+                            <p class="text-sm text-slate-400 mb-4">${ resource.description || 'No description provided.' }</p>
+                        </div>
+                        <div class="mt-auto">
+                            <a href="${ resource.link }" target="_blank" class="inline-flex items-center justify-center bg-indigo-600 hover:bg-indigo-500 text-white font-bold py-2 px-4 rounded-lg transition-colors duration-300 text-sm">
+                                Go to Resource
+                                <svg class="ml-2 -mr-1 w-4 h-4" fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg"><path d="M11 3a1 1 0 100 2h2.586l-6.293 6.293a1 1 0 101.414 1.414L15 6.414V9a1 1 0 102 0V4a1 1 0 00-1-1h-5z"></path><path d="M5 5a2 2 0 00-2 2v8a2 2 0 002 2h8a2 2 0 002-2v-3a1 1 0 10-2 0v3H5V7h3a1 1 0 000-2H5z"></path></svg>
+                            </a>
+                            ${ credentialsHtml }
+                        </div>
+                    `
+                    container.appendChild( resourceCard )
+                } )
+            } )
+
+            document.querySelectorAll( '.copy-btn' ).forEach( button =>
+            {
+                button.addEventListener( 'click', ( event ) =>
+                {
+                    const textToCopy = event.target.dataset.text
+                    copyToClipboard( textToCopy )
+                } )
+            } )
+        }
+
+        window.onload = renderResources;
+    </script>
+</body>
+
+</html>

infrastructure/grafana/grafana.ini

@@ -48,7 +48,7 @@ domain = localhost
 enforce_domain = false
 
 # The full public facing url
-root_url = %(protocol)s://%(domain)s:%(http_port)s/
+root_url = https://grafana.adnova.itqdev.xyz
 
 # Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
 serve_from_sub_path = false

infrastructure/grafana/provisioning/alerting/contact_points.yaml

@@ -0,0 +1,17 @@
+apiVersion: 1
+
+contactPoints:
+  - orgId: 1
+    name: Telegram
+    receivers:
+      - uid: aet1srtyc40lca
+        type: telegram
+        settings:
+          bottoken: 7797967907:AAGZuUzzuS4LLb525rDNY52Awc2tvpsLjd4
+          chatid: "-1002555823797"
+          disable_notification: false
+          disable_web_page_preview: false
+          message: '{{ template "telegram.default.message" . }}'
+          parse_mode: Markdown
+          protect_content: false
+        disableResolveMessage: false

infrastructure/grafana/provisioning/alerting/rules.yaml

@@ -0,0 +1,141 @@
+apiVersion: 1
+
+groups:
+  - orgId: 1
+    name: Default
+    folder: Backend
+    interval: 10s
+    rules:
+      - uid: aet1xbx1yaupsb
+        title: Backend p99 > 500 ms
+        condition: C
+        data:
+          - refId: A
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: prometheus
+            model:
+              editorMode: code
+              expr: |
+                histogram_quantile(
+                  0.99,
+                  sum(
+                    rate(
+                      caddy_http_request_duration_seconds_bucket{instance="proxy:2019",handler="reverse_proxy",host="proxy:8080",job="caddy"}[$__rate_interval]
+                    )
+                  ) by (le)
+                )
+              instant: true
+              intervalMs: 1000
+              legendFormat: __auto
+              maxDataPoints: 43200
+              range: false
+              refId: A
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              conditions:
+                - evaluator:
+                    params:
+                      - 0.5
+                    type: gte
+                  operator:
+                    type: and
+                  query:
+                    params:
+                      - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+              datasource:
+                type: __expr__
+                uid: __expr__
+              expression: A
+              intervalMs: 1000
+              maxDataPoints: 43200
+              refId: C
+              type: threshold
+        dashboardUid: e3a78c36-2f34-4ad6-81d5-284002896829
+        panelId: 32
+        noDataState: OK
+        execErrState: Error
+        for: 10s
+        keepFiringFor: 10s
+        annotations:
+          __dashboardUid__: e3a78c36-2f34-4ad6-81d5-284002896829
+          __panelId__: "32"
+          runbook_url: https://admin.adnova.itqdev.xyz
+          summary: p99 > 500 ms
+        isPaused: false
+        notification_settings:
+          receiver: Telegram
+  - orgId: 1
+    name: Default
+    folder: Postgres
+    interval: 10s
+    rules:
+      - uid: fet1txr4slywwe
+        title: "> 100 QPS on Postgresql"
+        condition: C
+        data:
+          - refId: A
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: prometheus
+            model:
+              editorMode: code
+              expr: |
+                sum(
+                  irate(pg_stat_database_xact_commit{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
+                )
+                + sum(
+                  irate(pg_stat_database_xact_rollback{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
+                )
+              instant: true
+              intervalMs: 1000
+              legendFormat: __auto
+              maxDataPoints: 43200
+              range: false
+              refId: A
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              conditions:
+                - evaluator:
+                    params:
+                      - 100
+                    type: gte
+                  operator:
+                    type: and
+                  query:
+                    params:
+                      - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+              datasource:
+                type: __expr__
+                uid: __expr__
+              expression: A
+              intervalMs: 1000
+              maxDataPoints: 43200
+              refId: C
+              type: threshold
+        dashboardUid: postgres-overview
+        panelId: 14
+        noDataState: NoData
+        execErrState: Error
+        for: 10s
+        keepFiringFor: 1m
+        annotations:
+          __dashboardUid__: postgres-overview
+          __panelId__: "14"
+          runbook_url: https://admin.adnova.itqdev.xyz
+          summary: Postgresql QPS exceeded 100
+        isPaused: false
+        notification_settings:
+          receiver: Telegram

infrastructure/grafana/provisioning/alerting/templates.yaml

@@ -0,0 +1,54 @@
+apiVersion: 1
+
+templates:
+  - orgId: 1
+    name: Telegram
+    template: |
+      {{ define "telegram.default.message" }} {{ if gt (len .Alerts.Firing) 0 }}
+      🔥🚨 *FIRE IN THE HOLE!* 🚨🔥
+      We've got *{{ len .Alerts.Firing }} firing alert(s)* that need your immediate attention!
+      {{ range .Alerts.Firing }}
+      ---
+      *Alert:* `{{ .Labels.alertname }}`
+      {{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
+      *Status:* 🔴 *FIRING* since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }}
+
+      {{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
+      {{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
+      *Labels:*
+      {{ range .Labels.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}
+      {{ if gt (len .Annotations) 0 }}*Annotations:*
+      {{ range .Annotations.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}{{ end }}
+
+      {{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
+      {{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
+      {{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }})
+      {{ end }}{{ if .SilenceURL }}🤫 [Silence Alert]({{ .SilenceURL }})
+      {{ end }}--- {{ end }} {{ end }}
+      {{ if gt (len .Alerts.Resolved) 0 }}
+      ✅🟢 *ALL CLEAR!* 🟢✅
+      Great news! *{{ len .Alerts.Resolved }} alert(s)* have been resolved.
+      {{ range .Alerts.Resolved }}
+      ---
+      *Alert:* `{{ .Labels.alertname }}`
+      {{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
+      *Status:* ✅ *RESOLVED* at {{ .EndsAt.Format "2006-01-02 15:04:05 MST" }} (was active since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }})
+
+      {{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
+      {{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
+      *Labels:*
+      {{ range .Labels.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}
+      {{ if gt (len .Annotations) 0 }}*Annotations:*
+      {{ range .Annotations.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}{{ end }}
+
+      {{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
+      {{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
+      {{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }}) {{ end }} {{ end }} {{ end }}
+      {{ if or (gt (len .Alerts.Firing) 0) (gt (len .Alerts.Resolved) 0) }}
+      🔔 *Grafana Alertmanager:* [View All Alerts]({{ template "__alertmanagerURL" . }}) 🔔
+      {{ end }}
+      {{ end }}

infrastructure/grafana/provisioning/dashboards/Backend/requests-by-view.json

@@ -0,0 +1,910 @@
+{
+    "annotations": {
+        "list": [
+            {
+                "builtIn": 1,
+                "datasource": {
+                    "type": "grafana",
+                    "uid": "-- Grafana --"
+                },
+                "enable": true,
+                "hide": true,
+                "iconColor": "rgba(0, 211, 255, 1)",
+                "name": "Annotations & Alerts",
+                "type": "dashboard"
+            }
+        ]
+    },
+    "description": "A dashboard that monitors Django which focuses on breaking down requests by view.",
+    "editable": true,
+    "fiscalYearStartMonth": 0,
+    "graphTooltip": 0,
+    "id": 12,
+    "links": [
+        {
+            "tags": ["backend"],
+            "targetBlank": true,
+            "title": "Backend Dashboards",
+            "type": "dashboards"
+        }
+    ],
+    "liveNow": true,
+    "panels": [
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 0
+            },
+            "id": 1,
+            "panels": [],
+            "title": "Summary",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "red"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 0.95
+                            },
+                            {
+                                "color": "green",
+                                "value": 0.99
+                            }
+                        ]
+                    },
+                    "unit": "percentunit"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 0,
+                "y": 1
+            },
+            "id": 2,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum(\n  rate(\n    django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\",\n        status!~\"[4-5].*\"\n      }[1w]\n    )\n) /\nsum(\n  rate(\n    django_http_responses_total_by_status_view_method_total{\n      namespace=~\"$namespace\",\n      job=~\"$job\",\n      view=\"$view\",\n      method=~\"$method\"\n    }[1w]\n  )\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Success Rate (non 4xx-5xx responses) [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 10
+                            },
+                            {
+                                "color": "red",
+                                "value": 100
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 6,
+                "y": 1
+            },
+            "id": 3,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum by (view) (\n  increase(\n    django_http_exceptions_total_by_view_total{\n      namespace=~\"$namespace\",\n      job=~\"$job\",\n      view=\"$view\",\n    }[1w]\n  ) > 0\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "HTTP Exceptions [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 1000
+                            },
+                            {
+                                "color": "red",
+                                "value": 2000
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 12,
+                "y": 1
+            },
+            "id": 4,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum (\n    rate (\n        django_http_requests_latency_seconds_by_view_method_bucket {\n          namespace=~\"$namespace\",\n          job=~\"$job\",\n          view=\"$view\",\n          method=~\"$method\"\n        }[$__range]\n    )\n  ) by (job, le)\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Average Request Latency (P50) [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 2500
+                            },
+                            {
+                                "color": "red",
+                                "value": 5000
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 18,
+                "y": 1
+            },
+            "id": 5,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum (\n    rate (\n        django_http_requests_latency_seconds_by_view_method_bucket {\n          namespace=~\"$namespace\",\n          job=~\"$job\",\n          view=\"$view\",\n          method=~\"$method\"\n        }[$__range]\n    )\n  ) by (job, le)\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Average Request Latency (P95) [1w]",
+            "type": "stat"
+        },
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 5
+            },
+            "id": 6,
+            "panels": [],
+            "title": "Request & Responses",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 0,
+                "y": 6
+            },
+            "id": 7,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_requests_total_by_view_transport_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "reqps",
+                    "refId": "A"
+                }
+            ],
+            "title": "Requests",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "percent"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "2xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "green",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "3xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "blue",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "4xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "yellow",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "5xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "red",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 12,
+                "y": 6
+            },
+            "id": 8,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"2.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "2xx",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"3.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "3xx",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"4.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "4xx",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"5.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "5xx",
+                    "refId": "D"
+                }
+            ],
+            "title": "Responses",
+            "type": "timeseries"
+        },
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 14
+            },
+            "id": 9,
+            "panels": [],
+            "title": "Latency & Status Codes",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "value"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 0,
+                "y": 15
+            },
+            "id": 10,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (namespace, job, view, status, method), 0.001\n)\n",
+                    "legendFormat": "{{ view }} / {{ status }} / {{ method }}",
+                    "refId": "A"
+                }
+            ],
+            "title": "Responses Status Codes",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 10,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 12,
+                "y": 15
+            },
+            "id": 11,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "50 - {{ view }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "95 - {{ view }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.99,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "99 - {{ view }}",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.999,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "99.9 - {{ view }}",
+                    "refId": "D"
+                }
+            ],
+            "title": "Request Latency",
+            "type": "timeseries"
+        }
+    ],
+    "preload": false,
+    "refresh": "5s",
+    "schemaVersion": 41,
+    "tags": ["backend"],
+    "templating": {
+        "list": [
+            {
+                "label": "Data source",
+                "name": "datasource",
+                "query": "prometheus",
+                "refresh": 1,
+                "type": "datasource"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Namespace",
+                "name": "namespace",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{}, namespace)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Job",
+                "name": "job",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\"}, job)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "View",
+                "name": "view",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view!~\"<unnamed view>|health_check:health_check_home|prometheus-django-metrics\"}, view)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": true,
+                "label": "Method",
+                "multi": true,
+                "name": "method",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view=~\"$view\"}, method)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            }
+        ]
+    },
+    "time": {
+        "from": "now-30m",
+        "to": "now"
+    },
+    "timepicker": {},
+    "timezone": "browser",
+    "title": "Requests / By View",
+    "uid": "backend-requests-by-view",
+    "version": 2,
+    "weekStart": "monday"
+}

infrastructure/grafana/provisioning/dashboards/Celery/by-task.json

@@ -0,0 +1,768 @@
+{
+    "annotations": {
+        "list": [
+            {
+                "builtIn": 1,
+                "datasource": {
+                    "type": "grafana",
+                    "uid": "-- Grafana --"
+                },
+                "enable": true,
+                "hide": true,
+                "iconColor": "rgba(0, 211, 255, 1)",
+                "name": "Annotations & Alerts",
+                "type": "dashboard"
+            }
+        ]
+    },
+    "description": "A dashboard that monitors Celery.",
+    "editable": true,
+    "fiscalYearStartMonth": 0,
+    "graphTooltip": 0,
+    "id": 2,
+    "links": [
+        {
+            "tags": ["celery"],
+            "targetBlank": true,
+            "title": "Celery Dashboards",
+            "type": "dashboards"
+        }
+    ],
+    "liveNow": true,
+    "panels": [
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 0
+            },
+            "id": 1,
+            "panels": [],
+            "title": "Tasks",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "align": "auto",
+                        "cellOptions": {
+                            "type": "auto"
+                        },
+                        "inspect": false
+                    },
+                    "mappings": [],
+                    "noValue": 0,
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "Success Rate"
+                        },
+                        "properties": [
+                            {
+                                "id": "unit",
+                                "value": "percentunit"
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 16,
+                "x": 0,
+                "y": 1
+            },
+            "id": 2,
+            "options": {
+                "cellHeight": "sm",
+                "footer": {
+                    "countRows": false,
+                    "enablePagination": true,
+                    "fields": "",
+                    "reducer": ["sum"],
+                    "show": false
+                },
+                "showHeader": true,
+                "sortBy": [
+                    {
+                        "desc": true,
+                        "displayName": "Succeeded"
+                    }
+                ]
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n/(sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n+sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n) > -1\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_sent_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "D"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_received_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "E"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_rejected_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "F"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_retried_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "G"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_revoked_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "H"
+                }
+            ],
+            "title": "Task Stats",
+            "transformations": [
+                {
+                    "id": "merge"
+                },
+                {
+                    "id": "organize",
+                    "options": {
+                        "excludeByName": {
+                            "Time": true
+                        },
+                        "indexByName": {
+                            "Value #A": 1,
+                            "Value #B": 2,
+                            "Value #C": 3,
+                            "Value #D": 4,
+                            "Value #E": 5,
+                            "Value #F": 6,
+                            "Value #G": 7,
+                            "Value #H": 8,
+                            "name": 0
+                        },
+                        "renameByName": {
+                            "Value #A": "Success Rate",
+                            "Value #B": "Succeeded",
+                            "Value #C": "Failed",
+                            "Value #D": "Sent",
+                            "Value #E": "Received",
+                            "Value #F": "Rejected",
+                            "Value #G": "Retried",
+                            "Value #H": "Revoked",
+                            "name": "Name"
+                        }
+                    }
+                }
+            ],
+            "type": "table"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "align": "auto",
+                        "cellOptions": {
+                            "type": "auto"
+                        },
+                        "inspect": false
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 8,
+                "x": 16,
+                "y": 1
+            },
+            "id": 3,
+            "options": {
+                "cellHeight": "sm",
+                "footer": {
+                    "countRows": false,
+                    "enablePagination": true,
+                    "fields": "",
+                    "reducer": ["sum"],
+                    "show": false
+                },
+                "showHeader": true,
+                "sortBy": [
+                    {
+                        "desc": true,
+                        "displayName": "Value"
+                    }
+                ]
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum (\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  ) by (name, exception) > 0\n)\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "A"
+                }
+            ],
+            "title": "Task Exceptions",
+            "transformations": [
+                {
+                    "id": "organize",
+                    "options": {
+                        "excludeByName": {
+                            "Time": true,
+                            "job": true
+                        },
+                        "indexByName": {
+                            "Value": 2,
+                            "exception": 1,
+                            "name": 0
+                        },
+                        "renameByName": {
+                            "exception": "Exception",
+                            "name": "Task"
+                        }
+                    }
+                }
+            ],
+            "type": "table"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 0,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 9
+            },
+            "id": 4,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Succeeded - {{ name }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Failed - {{ name }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_sent_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Sent - {{ name }}",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_received_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Received - {{ name }}",
+                    "refId": "D"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_retried_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Retried - {{ name }}",
+                    "refId": "E"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_revoked_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Revoked - {{ name }}",
+                    "refId": "F"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_rejected_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Rejected - {{ name }}",
+                    "refId": "G"
+                }
+            ],
+            "title": "Tasks Completed",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 0,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 17
+            },
+            "id": 5,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name, exception) > 0\n",
+                    "legendFormat": "{{ name }}/{{ exception }}",
+                    "refId": "A"
+                }
+            ],
+            "title": "Task Exceptions",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "spanNulls": false
+                    },
+                    "unit": "s"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P50"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "green",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P95"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "yellow",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P99"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "red",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 25
+            },
+            "id": 6,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "v11.1.0",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P50 - {{ name }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P95 - {{ name }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.99,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P99 - {{ name }}",
+                    "refId": "C"
+                }
+            ],
+            "title": "Tasks Runtime",
+            "type": "timeseries"
+        }
+    ],
+    "preload": false,
+    "refresh": "10s",
+    "schemaVersion": 41,
+    "tags": ["celery"],
+    "templating": {
+        "list": [
+            {
+                "label": "Data source",
+                "name": "datasource",
+                "query": "prometheus",
+                "refresh": 1,
+                "type": "datasource"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Namespace",
+                "name": "namespace",
+                "query": "label_values(celery_worker_up{}, namespace)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Job",
+                "name": "job",
+                "query": "label_values(celery_worker_up{namespace=\"$namespace\"}, job)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Queue Name",
+                "name": "queue_name",
+                "query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", name!~\"None\"}, queue_name)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Task",
+                "multi": true,
+                "name": "task",
+                "query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", queue_name=~\"$queue_name\", name!~\"None\"}, name)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            }
+        ]
+    },
+    "time": {
+        "from": "now-24h",
+        "to": "now"
+    },
+    "timepicker": {},
+    "timezone": "browser",
+    "title": "By Task",
+    "uid": "celery-tasks-by-task",
+    "version": 1,
+    "weekStart": "monday"
+}

infrastructure/grafana/provisioning/dashboards/providers.yaml

@@ -1,11 +1,13 @@
 apiVersion: 1
 
 providers:
-  - name: "celery"
+  - name: "default"
     orgId: 1
     folder: ""
     type: file
+    disableDeletion: false
     updateIntervalSeconds: 10
+    allowUiUpdates: false
     options:
       path: /etc/grafana/provisioning/dashboards
       foldersFromFilesStructure: true

infrastructure/grafana/provisioning/datasources/datasources.yaml

@@ -1,10 +1,29 @@
 apiVersion: 1
 
 datasources:
-  - name: Infinity
+  - name: Backend | Infinity
     type: yesoreyeram-infinity-datasource
     access: proxy
     orgId: 1
     uid: infinity
+    url: http://backend:8080
+    basicAuth: false
+    basicAuthUser: ""
+    withCredentials: false
     isDefault: false
     editable: false
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    orgId: 1
+    uid: prometheus
+    url: http://prometheus:9090
+    basicAuth: false
+    basicAuthUser: ""
+    withCredentials: false
+    isDefault: true
+    editable: false
+    jsonData:
+      httpMethod: POST
+      queryTimeout: 10s
+      timeInterval": 10s

infrastructure/pgadmin/.env.template

@@ -1,2 +1,4 @@
 PGADMIN_DEFAULT_EMAIL=admin@mail.com
-PGADMIN_DEFAULT_PASSWORD_FILE=/run/secrets/pgadmin_password
+PGADMIN_DEFAULT_PASSWORD=password
+PGADMIN_DISABLE_POSTFIX=True
+PGADMIN_REPLACE_SERVERS_ON_STARTUP=True

infrastructure/postgres/.env.template

@@ -1,3 +1,3 @@
 POSTGRES_DB=postgres
 POSTGRES_USER=postgres
-POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
+POSTGRES_PASSWORD=postgres

infrastructure/prometheus/prometheus.yaml

@@ -1,7 +1,7 @@
 # Global config
 global:
-  scrape_interval: 10s
+  scrape_interval: 5s
-  scrape_timeout: 10s
+  scrape_timeout: 5s
   evaluation_interval: 10m
   external_labels:
     environment: local
@@ -10,24 +10,31 @@ global:
 scrape_configs:
   # Prometheus
   - job_name: prometheus
-    scrape_interval: 5s
     static_configs:
       - targets: ["localhost:9090"]
 
   # Postgres
   - job_name: postgres
-    scrape_interval: 10s
     static_configs:
       - targets: ["postgres-exporter:9187"]
 
   # Redis
   - job_name: redis
-    scrape_interval: 10s
     static_configs:
       - targets: ["redis-exporter:9121"]
 
   # Celery
   - job_name: celery
-    scrape_interval: 30s
+    scrape_interval: 15s
     static_configs:
       - targets: ["celery-exporter:9808"]
+
+  # Backend
+  - job_name: backend
+    static_configs:
+      - targets: ["backend:8080"]
+
+  # Caddy
+  - job_name: caddy
+    static_configs:
+      - targets: ["proxy:2019"]

services/loadtest/static/index.html

@@ -263,7 +263,8 @@
 
         function setupWebSocket ()
         {
-            const wsUrl = 'ws://' + window.location.host + '/ws'
+            const protocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://'
+            const wsUrl = protocol + window.location.host + '/ws'
             ws = new WebSocket( wsUrl )
 
             ws.onopen = () => console.log( 'WebSocket connected' )

tests/e2e/.env.template

@@ -1,3 +1,3 @@
 # Below all environment variables and default values
 
-BACKEND_BASE_URL=http://127.0.0.1:8080
+BACKEND_BASE_URL=http://127.0.0.1:13240

tests/e2e/README.md

@@ -11,7 +11,7 @@ Ensure you have the following installed on your system:
 
 ## Warning
 
-Please note that containers will use ports from 13241 to 13246 and 8080, so there is must be no listeners on this ports range.
+Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
 
 ## Setup
 

tests/e2e/conftest.py

@@ -9,7 +9,7 @@ from dotenv import load_dotenv
 
 load_dotenv()
 
-BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:8080")
+BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:13240")
 
 
 @pytest.fixture(scope="session", autouse=True)
@@ -20,6 +20,16 @@ def docker_compose() -> Generator[None]:
         args=[
             "docker",
             "compose",
+            "--profile",
+            "loadtest",
+            "--profile",
+            "minio",
+            "--profile",
+            "observability",
+            "--profile",
+            "proxy",
+            "--profile",
+            "telegram_bot",
             "down",
         ],
         check=True,
@@ -31,6 +41,8 @@ def docker_compose() -> Generator[None]:
             "compose",
             "--project-name",
             "adnova-testing",
+            "--profile",
+            "minio",
             "up",
             "-d",
             "--build",
@@ -50,6 +62,8 @@ def docker_compose() -> Generator[None]:
             "compose",
             "--project-name",
             "adnova-testing",
+            "--profile",
+            "minio",
             "down",
             "-v",
         ],