fix(grafana): fixed false-positive NoData state

btw, they are incorrect

README.md

@@ -32,7 +32,7 @@ Table Report:
 
 #### Warning
 
-Please note that containers will use ports from 13241 to 13245 and 8080, so there is must be no listeners on this ports range.
+Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
 
 #### Configure
 
@@ -66,7 +66,7 @@ docker compose up -d --build
 
 #### Structure
 
-- **backend**: [127.0.0.1:8080](http://127.0.0.1:8080) -> `8080`
+- **backend**: [127.0.0.1:13240](http://127.0.0.1:13240) -> `8080`
   - Depends on: `postgres`, `redis`, `minio`, `backend-initdb`
 - **backend-initdb**
   - Depends on: `postgres`, `redis`, `minio`
@@ -139,7 +139,7 @@ You may say: "For what we need a lot of complex technologies for now". I have an
 
 ### Restful API
 
-API Base endpoint when deployed with default docker compose: [127.0.0.1:8080](http://127.0.0.1:8080), also see [docs](#openapi-docs).
+API Base endpoint when deployed with default docker compose: [127.0.0.1:13240](http://127.0.0.1:13240), also see [docs](#openapi-docs).
 
 ### Admin panel
 
@@ -152,9 +152,13 @@ Link: [t.me/adnova_bot](https://t.me/adnova_bot)
 Basic commands:
 
 `/start` - Start the bot and authenticate as advertiser
+
 `/help` - Get list of all commands
+
 `/campaigns` - Manage advertiser campaigns (only after authentication)
+
 `/statistics` - See advertiser overall statistics (only after authentication)
+
 `/logout` - Logout of current advertiser account (only after authentication)
 
 See [this](#telegram-bot-1).
@@ -234,9 +238,9 @@ Moderation implemented via report system. Client goes to `/report` ([see OpenAPI
 Also admin user (whose credentials specified lower) can add new staff members and even create a specified group for them (this is built-in django capabilities).
 Report has four states: Sent, Under review, Took action and Skipped. Admin panel has filtration by states and by flagged by llm status.
 
-Admin panel when deployed with docker compose (by default): [localhost:8080/admin/](http://localhost:8080/admin/)
+Admin panel when deployed with docker compose (by default): [localhost:13240/admin/](http://localhost:13240/admin/)
 
-Reports list when deployed with docker compose (requires authentication): [localhost:8080/admin/campaign/campaignreport/](http://localhost:8080/admin/campaign/campaignreport/)
+Reports list when deployed with docker compose (requires authentication): [localhost:13240/admin/campaign/campaignreport/](http://localhost:13240/admin/campaign/campaignreport/)
 
 Default username: `admin`
 
@@ -276,11 +280,11 @@ Demonstration:
 
 ### OpenAPI docs
 
-When deployed with default docker compose: [localhost:8080/docs](http://localhost:8080/docs)
+When deployed with default docker compose: [localhost:13240/docs](http://localhost:13240/docs)
 
 ### Healthcheck endpoint
 
-When deployed with default docker compose: [localhost:8080/health](http://localhost:8080/health)
+When deployed with default docker compose: [localhost:13240/health](http://localhost:13240/health)
 
 Lets developers easily understand and identify problem and users check services health.
 

compose.yaml

@@ -5,6 +5,9 @@ services:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     depends_on:
       backend-initdb:
         restart: false
@@ -18,10 +21,6 @@ services:
         restart: false
         condition: service_healthy
         required: true
-      minio:
-        restart: false
-        condition: service_healthy
-        required: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
@@ -30,16 +29,20 @@ services:
     ports:
       - name: web
         target: 8080
-        published: 8080
+        published: 13240
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
     restart: unless-stopped
+    shm_size: 4mb
 
   backend-initdb:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     command: ./scripts/initdb
     depends_on:
       postgres:
@@ -50,27 +53,27 @@ services:
         restart: false
         condition: service_healthy
         required: true
-      minio:
-        restart: false
-        condition: service_healthy
-        required: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
       - path: ./infrastructure/backend/.env
         required: false
+    shm_size: 4mb
 
   backend-staticfiles:
     build:
       context: ./services/backend
       dockerfile: Dockerfile.staticfiles
+      tags:
+        - adnova-backend-staticfiles:latest
+      pull: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
       - path: ./infrastructure/backend/.env
         required: false
     healthcheck:
-      test: ["CMD", "service", "nginx", "status", "||", " exit 1"]
+      test: ["CMD-SHELL", "nginx", "-t", "||", "exit 1"]
       interval: 1m30s
       timeout: 5s
       start_period: 5s
@@ -84,11 +87,15 @@ services:
         protocol: tcp
         app_protocol: http
     restart: unless-stopped
+    shm_size: 4mb
 
   backend-celery-worker:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     command: celery -A config worker -l INFO
     depends_on:
       redis:
@@ -108,6 +115,7 @@ services:
       start_period: 10s
       start_interval: 2s
     restart: unless-stopped
+    shm_size: 4mb
 
   celery-exporter:
     image: docker.io/danihodovic/celery-exporter:0.12.2
@@ -122,7 +130,10 @@ services:
         required: true
       - path: ./infrastructure/celery-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
+    shm_size: 4mb
 
   telegram_bot:
     build:
@@ -130,6 +141,7 @@ services:
       dockerfile: Dockerfile
       tags:
         - adnova-telegram_bot:latest
+      pull: true
     depends_on:
       backend:
         restart: false
@@ -148,7 +160,10 @@ services:
         required: true
       - path: ./infrastructure/telegram_bot/.env
         required: false
+    profiles:
+      - telegram_bot
     restart: unless-stopped
+    shm_size: 4mb
 
   redis:
     image: docker.io/redis:7-alpine3.21
@@ -174,6 +189,7 @@ services:
       - type: volume
         source: redis_data
         target: /data
+        read_only: false
 
   redis-exporter:
     image: docker.io/oliver006/redis_exporter:v1.74.0-alpine
@@ -187,6 +203,8 @@ services:
         required: true
       - path: ./infrastructure/redis-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
 
@@ -201,7 +219,7 @@ services:
       - path: ./infrastructure/postgres/.env
         required: false
     healthcheck:
-      test: ["CMD", "pg_isready"]
+      test: ["CMD", "pg_isready", "-U", "postgres"]
       interval: 1m30s
       timeout: 5s
       start_period: 5s
@@ -209,14 +227,12 @@ services:
       retries: 5
     oom_kill_disable: true
     restart: unless-stopped
-    secrets:
-      - source: postgres_password
-        target: /run/secrets/postgres_password
     shm_size: 128mb
     volumes:
       - type: volume
         source: postgres_data
         target: /var/lib/postgresql/data
+        read_only: false
 
   postgres-exporter:
     image: quay.io/prometheuscommunity/postgres-exporter:v0.17.1
@@ -230,11 +246,13 @@ services:
         required: true
       - path: ./infrastructure/postgres-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
 
   pgadmin:
-    image: docker.io/dpage/pgadmin4:9.5
+    image: docker.io/dpage/pgadmin4:9.6
     configs:
       - source: pgadmin_servers_config
         target: /pgadmin4/servers.json
@@ -262,22 +280,22 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
-    secrets:
-      - source: pgadmin_password
-        target: /run/secrets/pgadmin_password
     shm_size: 4mb
     volumes:
       - type: volume
         source: pgadmin_data
         target: /var/lib/pgadmin
+        read_only: false
 
   grafana:
     image: docker.io/grafana/grafana-oss:12.0.2
+    entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
     configs:
       - source: grafana_config
         target: /usr/share/grafana/conf/defaults.ini
-    entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
     healthcheck:
       test: ["CMD", "wget", "-O", "-", "http://localhost:3000/api/health"]
       interval: 1m30s
@@ -292,18 +310,23 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
     volumes:
       - type: volume
         source: grafana_data
         target: /var/lib/grafana
+        read_only: false
       - type: bind
         source: ./infrastructure/grafana/provisioning
         target: /etc/grafana/provisioning
+        read_only: true
       - type: bind
         source: ./infrastructure/grafana/scripts
         target: /etc/grafana/scripts
+        read_only: true
 
   prometheus:
     image: docker.io/prom/prometheus:v3.5.0
@@ -325,12 +348,15 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
     volumes:
       - type: volume
         source: prometheus_data
         target: /prometheus
+        read_only: false
 
   minio:
     image: docker.io/minio/minio:RELEASE.2025-07-18T21-56-31Z
@@ -360,11 +386,106 @@ services:
         host_ip: 127.0.0.1
         protocol: tcp
         app_protocol: http
+    profiles:
+      - minio
     restart: unless-stopped
+    shm_size: 4mb
     volumes:
       - type: volume
         source: minio_data
         target: /data
+        read_only: false
+
+  zipkin:
+    image: docker.io/openzipkin/zipkin:3
+    healthcheck:
+      test: ["CMD", "wget", "-O", "-", "http://localhost:9411/health"]
+      interval: 1m30s
+      timeout: 5s
+      start_period: 5s
+      start_interval: 2s
+      retries: 5
+    ports:
+      - name: web
+        target: 9411
+        published: 13247
+        host_ip: 127.0.0.1
+        protocol: tcp
+        app_protocol: http
+    profiles:
+      - observability
+    restart: unless-stopped
+    shm_size: 4mb
+
+  loadtest:
+    build:
+      context: ./services/loadtest
+      dockerfile: Dockerfile
+      tags:
+        - adnova-loadtest:latest
+    depends_on:
+      backend:
+        restart: false
+        condition: service_healthy
+        required: true
+    env_file:
+      - path: ./infrastructure/loadtest/.env.template
+        required: true
+      - path: ./infrastructure/loadtest/.env
+        required: false
+    ports:
+      - name: web
+        target: 5001
+        published: 13248
+        host_ip: 127.0.0.1
+        protocol: tcp
+        app_protocol: http
+    profiles:
+      - loadtest
+    restart: unless-stopped
+    shm_size: 4mb
+
+  proxy:
+    image: docker.io/caddy:2-alpine
+    configs:
+      - source: caddy_config
+        target: /etc/caddy/Caddyfile
+    ports:
+      - name: http
+        target: 80
+        published: 80
+        host_ip: 0.0.0.0
+        protocol: tcp
+        app_protocol: http
+      - name: https
+        target: 443
+        published: 443
+        host_ip: 0.0.0.0
+        protocol: tcp
+        app_protocol: http
+      - name: http3
+        target: 443
+        published: 443
+        host_ip: 0.0.0.0
+        protocol: udp
+        app_protocol: http
+    profiles:
+      - proxy
+    restart: unless-stopped
+    shm_size: 4mb
+    volumes:
+      - type: volume
+        source: caddy_data
+        target: /data
+        read_only: false
+      - type: volume
+        source: caddy_config
+        target: /config
+        read_only: false
+      - type: bind
+        source: ./infrastructure/caddy/static
+        target: /var/www
+        read_only: true
 
 volumes:
   redis_data:
@@ -373,6 +494,8 @@ volumes:
   grafana_data:
   prometheus_data:
   minio_data:
+  caddy_data:
+  caddy_config:
 
 configs:
   redis_config:
@@ -385,9 +508,5 @@ configs:
     file: ./infrastructure/grafana/grafana.ini
   prometheus_config:
     file: ./infrastructure/prometheus/prometheus.yaml
-
+  caddy_config:
-secrets:
+    file: ./infrastructure/caddy/Caddyfile
-  postgres_password:
-    file: ./infrastructure/postgres/password
-  pgadmin_password:
-    file: ./infrastructure/pgadmin/password

infrastructure/backend/.env.template

@@ -1,4 +1,4 @@
-DJANGO_SECRET_KEY=secretees
+DJANGO_SECRET_KEY=very_insecure_key
 DJANGO_DEBUG=False
 DJANGO_ALLOWED_HOSTS=*
 DJANGO_CSRF_TRUSTED_ORIGINS=http://localhost,http://127.0.0.1
@@ -21,3 +21,7 @@ MINIO_ENDPOINT=minio:9000
 MINIO_CUSTOM_ENDPOINT_URL=http://127.0.0.1:13244
 MINIO_ACCESS_KEY=admin
 MINIO_SECRET_KEY=password
+
+OTEL_METRICS_EXPORTER=none
+OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://zipkin:9411/api/v2/spans
+OTEL_TRACES_EXPORTER=zipkin_json

infrastructure/caddy/Caddyfile

@@ -0,0 +1,68 @@
+{
+	admin :2019
+
+	metrics {
+		per_host
+	}
+}
+
+(basic-auth) {
+	basic_auth {
+		admin $2a$14$2zQilpLka2h8Sn1mmOLAAezwDN8Zy8Ta36WECk4qt5MTn3CWksR0m
+	}
+}
+
+adnova.itqdev.xyz {
+	@healthPath path /health /health/*
+	handle @healthPath {
+		import basic-auth
+		reverse_proxy http://backend:8080
+	}
+
+	handle_path /static/* {
+		reverse_proxy http://backend-staticfiles:80
+	}
+
+	reverse_proxy http://backend:8080
+}
+
+:8080 {
+	reverse_proxy http://backend:8080
+}
+
+admin.adnova.itqdev.xyz {
+	import basic-auth
+
+	root * /var/www/admin
+	file_server
+}
+
+loadtest.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://loadtest:5001
+}
+
+grafana.adnova.itqdev.xyz {
+	reverse_proxy http://grafana:3000
+}
+
+pgadmin.adnova.itqdev.xyz {
+	reverse_proxy http://pgadmin:80
+}
+
+zipkin.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://zipkin:9411
+}
+
+prometheus.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://prometheus:9090
+}
+
+minio.adnova.itqdev.xyz {
+	reverse_proxy http://minio:9000
+}

infrastructure/caddy/static/admin/index.html

@@ -0,0 +1,209 @@
+<!DOCTYPE html>
+<html lang="en" class="dark">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>AdNova Admin Resources</title>
+    <script src="https://cdn.tailwindcss.com"></script>
+    <style>
+        body {
+            font-family: 'Inter', sans-serif;
+        }
+
+        @import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');
+    </style>
+</head>
+
+<body class="bg-slate-900 text-gray-300">
+    <div class="container mx-auto p-4 md:p-8">
+        <header class="mb-8">
+            <h1 class="text-4xl font-bold text-white tracking-tight">AdNova Admin Resources Dashboard</h1>
+            <p class="text-indigo-400">Quick access to essential tools and credentials.</p>
+        </header>
+
+        <div id="resources-container" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
+        </div>
+    </div>
+
+    <script>
+        const resources = [
+            {
+                category: "Development Tools",
+                items: [
+                    {
+                        name: "Gitea Repository",
+                        link: "https://git.itqdev.xyz/PROD.2025/AdNova",
+                        description: "Main monorepo on Gitea."
+                    },
+                    {
+                        name: "Rest API Docs",
+                        link: "https://adnova.itqdev.xyz/docs",
+                        description: "Rest API Docs on Swagger."
+                    },
+                    {
+                        name: "Django Admin",
+                        link: "https://adnova.itqdev.xyz/admin/",
+                        username: "admin",
+                        password: "admin",
+                        description: "Django Admin panel."
+                    },
+                    {
+                        name: "Loadtest",
+                        link: "https://loadtest.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Stress test utility with multiple profiles and ability to load mocks."
+                    }
+                ]
+            },
+            {
+                category: "Monitoring",
+                items: [
+                    {
+                        name: "Grafana",
+                        link: "https://grafana.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "proooooood",
+                        description: "Real-time system metrics and dashboards."
+                    },
+                    {
+                        name: "Prometheus",
+                        link: "https://prometheus.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Real-time system metrics and scrape targets."
+                    },
+                    {
+                        name: "Zipkin",
+                        link: "https://zipkin.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Latest traces from production."
+                    },
+                    {
+                        name: "Django Silk",
+                        link: "https://adnova.itqdev.xyz/silk/",
+                        username: "admin",
+                        password: "admin",
+                        description: "App profiles and performance info in production, need to authenticate first as staff in Django Admin."
+                    },
+                    {
+                        name: "Adnova Alerts",
+                        link: "https://t.me/adnova_alerts",
+                        description: "Telegram channel where alerts are posted."
+                    },
+                    {
+                        name: "Healthcheck",
+                        link: "https://adnova.itqdev.xyz/health",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Summary of integration status between services."
+                    },
+                    {
+                        name: "Uptime-Kuma Dashboard",
+                        link: "https://status.adnova.itqdev.xyz",
+                        description: "External monitoring of AdNova."
+                    }
+                ]
+            },
+            {
+                category: "Databases",
+                items: [
+                    {
+                        name: "Pgadmin",
+                        link: "https://pgadmin.adnova.itqdev.xyz",
+                        username: "admin@mail.com",
+                        password: "password",
+                        description: "Access to production database."
+                    }
+                ]
+            }
+        ]
+
+        function copyToClipboard ( text )
+        {
+            const textarea = document.createElement( 'textarea' )
+            textarea.value = text
+            document.body.appendChild( textarea )
+            textarea.select()
+            document.execCommand( 'copy' )
+            document.body.removeChild( textarea )
+            alert( 'Copied to clipboard!' )
+        }
+
+        function renderResources ()
+        {
+            const container = document.getElementById( 'resources-container' )
+            container.innerHTML = ''
+
+            resources.forEach( categoryData =>
+            {
+                const categorySection = document.createElement( 'div' )
+                categorySection.className = 'col-span-full mb-4'
+                categorySection.innerHTML = `
+                    <h2 class="text-2xl font-semibold text-white mb-4 border-b border-slate-700 pb-2">
+                        ${ categoryData.category }
+                    </h2>
+                `
+                container.appendChild( categorySection )
+
+                categoryData.items.forEach( resource =>
+                {
+                    const resourceCard = document.createElement( 'div' )
+                    resourceCard.className = 'bg-slate-800 rounded-lg p-6 shadow-lg flex flex-col justify-between'
+
+                    let credentialsHtml = ''
+                    if ( resource.username || resource.password )
+                    {
+                        credentialsHtml = `
+                            <div class="mt-4 pt-4 border-t border-slate-700">
+                                <h4 class="text-sm font-medium text-slate-400 mb-2">Credentials:</h4>
+                                ${ resource.username ? `
+                                    <div class="flex items-center text-sm mb-2">
+                                        <span class="font-semibold text-slate-300 w-24">Username:</span>
+                                        <span id="user-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.username }</span>
+                                        <button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.username }">Copy</button>
+                                    </div>` : '' }
+                                ${ resource.password ? `
+                                    <div class="flex items-center text-sm">
+                                        <span class="font-semibold text-slate-300 w-24">Password:</span>
+                                        <span id="pass-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.password }</span>
+                                        <button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.password }">Copy</button>
+                                    </div>` : '' }
+                            </div>
+                        `
+                    }
+
+                    resourceCard.innerHTML = `
+                        <div>
+                            <h3 class="text-xl font-semibold text-white mb-2">${ resource.name }</h3>
+                            <p class="text-sm text-slate-400 mb-4">${ resource.description || 'No description provided.' }</p>
+                        </div>
+                        <div class="mt-auto">
+                            <a href="${ resource.link }" target="_blank" class="inline-flex items-center justify-center bg-indigo-600 hover:bg-indigo-500 text-white font-bold py-2 px-4 rounded-lg transition-colors duration-300 text-sm">
+                                Go to Resource
+                                <svg class="ml-2 -mr-1 w-4 h-4" fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg"><path d="M11 3a1 1 0 100 2h2.586l-6.293 6.293a1 1 0 101.414 1.414L15 6.414V9a1 1 0 102 0V4a1 1 0 00-1-1h-5z"></path><path d="M5 5a2 2 0 00-2 2v8a2 2 0 002 2h8a2 2 0 002-2v-3a1 1 0 10-2 0v3H5V7h3a1 1 0 000-2H5z"></path></svg>
+                            </a>
+                            ${ credentialsHtml }
+                        </div>
+                    `
+                    container.appendChild( resourceCard )
+                } )
+            } )
+
+            document.querySelectorAll( '.copy-btn' ).forEach( button =>
+            {
+                button.addEventListener( 'click', ( event ) =>
+                {
+                    const textToCopy = event.target.dataset.text
+                    copyToClipboard( textToCopy )
+                } )
+            } )
+        }
+
+        window.onload = renderResources;
+    </script>
+</body>
+
+</html>

infrastructure/grafana/grafana.ini

@@ -48,7 +48,7 @@ domain = localhost
 enforce_domain = false
 
 # The full public facing url
-root_url = %(protocol)s://%(domain)s:%(http_port)s/
+root_url = https://grafana.adnova.itqdev.xyz
 
 # Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
 serve_from_sub_path = false

infrastructure/grafana/provisioning/alerting/contact_points.yaml

@@ -0,0 +1,17 @@
+apiVersion: 1
+
+contactPoints:
+  - orgId: 1
+    name: Telegram
+    receivers:
+      - uid: aet1srtyc40lca
+        type: telegram
+        settings:
+          bottoken: 7797967907:AAGZuUzzuS4LLb525rDNY52Awc2tvpsLjd4
+          chatid: "-1002555823797"
+          disable_notification: false
+          disable_web_page_preview: false
+          message: '{{ template "telegram.default.message" . }}'
+          parse_mode: Markdown
+          protect_content: false
+        disableResolveMessage: false

infrastructure/grafana/provisioning/alerting/rules.yaml

@@ -0,0 +1,141 @@
+apiVersion: 1
+
+groups:
+  - orgId: 1
+    name: Default
+    folder: Backend
+    interval: 10s
+    rules:
+      - uid: aet1xbx1yaupsb
+        title: Backend p99 > 500 ms
+        condition: C
+        data:
+          - refId: A
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: prometheus
+            model:
+              editorMode: code
+              expr: |
+                histogram_quantile(
+                  0.99,
+                  sum(
+                    rate(
+                      caddy_http_request_duration_seconds_bucket{instance="proxy:2019",handler="reverse_proxy",host="proxy:8080",job="caddy"}[$__rate_interval]
+                    )
+                  ) by (le)
+                )
+              instant: true
+              intervalMs: 1000
+              legendFormat: __auto
+              maxDataPoints: 43200
+              range: false
+              refId: A
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              conditions:
+                - evaluator:
+                    params:
+                      - 0.5
+                    type: gte
+                  operator:
+                    type: and
+                  query:
+                    params:
+                      - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+              datasource:
+                type: __expr__
+                uid: __expr__
+              expression: A
+              intervalMs: 1000
+              maxDataPoints: 43200
+              refId: C
+              type: threshold
+        dashboardUid: e3a78c36-2f34-4ad6-81d5-284002896829
+        panelId: 32
+        noDataState: OK
+        execErrState: Error
+        for: 10s
+        keepFiringFor: 10s
+        annotations:
+          __dashboardUid__: e3a78c36-2f34-4ad6-81d5-284002896829
+          __panelId__: "32"
+          runbook_url: https://admin.adnova.itqdev.xyz
+          summary: p99 > 500 ms
+        isPaused: false
+        notification_settings:
+          receiver: Telegram
+  - orgId: 1
+    name: Default
+    folder: Postgres
+    interval: 10s
+    rules:
+      - uid: fet1txr4slywwe
+        title: "> 100 QPS on Postgresql"
+        condition: C
+        data:
+          - refId: A
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: prometheus
+            model:
+              editorMode: code
+              expr: |
+                sum(
+                  irate(pg_stat_database_xact_commit{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
+                )
+                + sum(
+                  irate(pg_stat_database_xact_rollback{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
+                )
+              instant: true
+              intervalMs: 1000
+              legendFormat: __auto
+              maxDataPoints: 43200
+              range: false
+              refId: A
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              conditions:
+                - evaluator:
+                    params:
+                      - 100
+                    type: gte
+                  operator:
+                    type: and
+                  query:
+                    params:
+                      - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+              datasource:
+                type: __expr__
+                uid: __expr__
+              expression: A
+              intervalMs: 1000
+              maxDataPoints: 43200
+              refId: C
+              type: threshold
+        dashboardUid: postgres-overview
+        panelId: 14
+        noDataState: NoData
+        execErrState: Error
+        for: 10s
+        keepFiringFor: 1m
+        annotations:
+          __dashboardUid__: postgres-overview
+          __panelId__: "14"
+          runbook_url: https://admin.adnova.itqdev.xyz
+          summary: Postgresql QPS exceeded 100
+        isPaused: false
+        notification_settings:
+          receiver: Telegram

infrastructure/grafana/provisioning/alerting/templates.yaml

@@ -0,0 +1,54 @@
+apiVersion: 1
+
+templates:
+  - orgId: 1
+    name: Telegram
+    template: |
+      {{ define "telegram.default.message" }} {{ if gt (len .Alerts.Firing) 0 }}
+      🔥🚨 *FIRE IN THE HOLE!* 🚨🔥
+      We've got *{{ len .Alerts.Firing }} firing alert(s)* that need your immediate attention!
+      {{ range .Alerts.Firing }}
+      ---
+      *Alert:* `{{ .Labels.alertname }}`
+      {{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
+      *Status:* 🔴 *FIRING* since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }}
+
+      {{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
+      {{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
+      *Labels:*
+      {{ range .Labels.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}
+      {{ if gt (len .Annotations) 0 }}*Annotations:*
+      {{ range .Annotations.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}{{ end }}
+
+      {{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
+      {{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
+      {{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }})
+      {{ end }}{{ if .SilenceURL }}🤫 [Silence Alert]({{ .SilenceURL }})
+      {{ end }}--- {{ end }} {{ end }}
+      {{ if gt (len .Alerts.Resolved) 0 }}
+      ✅🟢 *ALL CLEAR!* 🟢✅
+      Great news! *{{ len .Alerts.Resolved }} alert(s)* have been resolved.
+      {{ range .Alerts.Resolved }}
+      ---
+      *Alert:* `{{ .Labels.alertname }}`
+      {{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
+      *Status:* ✅ *RESOLVED* at {{ .EndsAt.Format "2006-01-02 15:04:05 MST" }} (was active since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }})
+
+      {{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
+      {{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
+      *Labels:*
+      {{ range .Labels.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}
+      {{ if gt (len .Annotations) 0 }}*Annotations:*
+      {{ range .Annotations.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}{{ end }}
+
+      {{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
+      {{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
+      {{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }}) {{ end }} {{ end }} {{ end }}
+      {{ if or (gt (len .Alerts.Firing) 0) (gt (len .Alerts.Resolved) 0) }}
+      🔔 *Grafana Alertmanager:* [View All Alerts]({{ template "__alertmanagerURL" . }}) 🔔
+      {{ end }}
+      {{ end }}

infrastructure/grafana/provisioning/dashboards/Backend/requests-by-view.json

@@ -0,0 +1,910 @@
+{
+    "annotations": {
+        "list": [
+            {
+                "builtIn": 1,
+                "datasource": {
+                    "type": "grafana",
+                    "uid": "-- Grafana --"
+                },
+                "enable": true,
+                "hide": true,
+                "iconColor": "rgba(0, 211, 255, 1)",
+                "name": "Annotations & Alerts",
+                "type": "dashboard"
+            }
+        ]
+    },
+    "description": "A dashboard that monitors Django which focuses on breaking down requests by view.",
+    "editable": true,
+    "fiscalYearStartMonth": 0,
+    "graphTooltip": 0,
+    "id": 12,
+    "links": [
+        {
+            "tags": ["backend"],
+            "targetBlank": true,
+            "title": "Backend Dashboards",
+            "type": "dashboards"
+        }
+    ],
+    "liveNow": true,
+    "panels": [
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 0
+            },
+            "id": 1,
+            "panels": [],
+            "title": "Summary",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "red"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 0.95
+                            },
+                            {
+                                "color": "green",
+                                "value": 0.99
+                            }
+                        ]
+                    },
+                    "unit": "percentunit"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 0,
+                "y": 1
+            },
+            "id": 2,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum(\n  rate(\n    django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\",\n        status!~\"[4-5].*\"\n      }[1w]\n    )\n) /\nsum(\n  rate(\n    django_http_responses_total_by_status_view_method_total{\n      namespace=~\"$namespace\",\n      job=~\"$job\",\n      view=\"$view\",\n      method=~\"$method\"\n    }[1w]\n  )\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Success Rate (non 4xx-5xx responses) [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 10
+                            },
+                            {
+                                "color": "red",
+                                "value": 100
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 6,
+                "y": 1
+            },
+            "id": 3,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum by (view) (\n  increase(\n    django_http_exceptions_total_by_view_total{\n      namespace=~\"$namespace\",\n      job=~\"$job\",\n      view=\"$view\",\n    }[1w]\n  ) > 0\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "HTTP Exceptions [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 1000
+                            },
+                            {
+                                "color": "red",
+                                "value": 2000
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 12,
+                "y": 1
+            },
+            "id": 4,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum (\n    rate (\n        django_http_requests_latency_seconds_by_view_method_bucket {\n          namespace=~\"$namespace\",\n          job=~\"$job\",\n          view=\"$view\",\n          method=~\"$method\"\n        }[$__range]\n    )\n  ) by (job, le)\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Average Request Latency (P50) [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 2500
+                            },
+                            {
+                                "color": "red",
+                                "value": 5000
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 18,
+                "y": 1
+            },
+            "id": 5,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum (\n    rate (\n        django_http_requests_latency_seconds_by_view_method_bucket {\n          namespace=~\"$namespace\",\n          job=~\"$job\",\n          view=\"$view\",\n          method=~\"$method\"\n        }[$__range]\n    )\n  ) by (job, le)\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Average Request Latency (P95) [1w]",
+            "type": "stat"
+        },
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 5
+            },
+            "id": 6,
+            "panels": [],
+            "title": "Request & Responses",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 0,
+                "y": 6
+            },
+            "id": 7,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_requests_total_by_view_transport_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "reqps",
+                    "refId": "A"
+                }
+            ],
+            "title": "Requests",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "percent"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "2xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "green",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "3xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "blue",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "4xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "yellow",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "5xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "red",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 12,
+                "y": 6
+            },
+            "id": 8,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"2.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "2xx",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"3.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "3xx",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"4.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "4xx",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"5.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "5xx",
+                    "refId": "D"
+                }
+            ],
+            "title": "Responses",
+            "type": "timeseries"
+        },
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 14
+            },
+            "id": 9,
+            "panels": [],
+            "title": "Latency & Status Codes",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "value"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 0,
+                "y": 15
+            },
+            "id": 10,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (namespace, job, view, status, method), 0.001\n)\n",
+                    "legendFormat": "{{ view }} / {{ status }} / {{ method }}",
+                    "refId": "A"
+                }
+            ],
+            "title": "Responses Status Codes",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 10,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 12,
+                "y": 15
+            },
+            "id": 11,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "50 - {{ view }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "95 - {{ view }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.99,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "99 - {{ view }}",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.999,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "99.9 - {{ view }}",
+                    "refId": "D"
+                }
+            ],
+            "title": "Request Latency",
+            "type": "timeseries"
+        }
+    ],
+    "preload": false,
+    "refresh": "5s",
+    "schemaVersion": 41,
+    "tags": ["backend"],
+    "templating": {
+        "list": [
+            {
+                "label": "Data source",
+                "name": "datasource",
+                "query": "prometheus",
+                "refresh": 1,
+                "type": "datasource"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Namespace",
+                "name": "namespace",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{}, namespace)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Job",
+                "name": "job",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\"}, job)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "View",
+                "name": "view",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view!~\"<unnamed view>|health_check:health_check_home|prometheus-django-metrics\"}, view)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": true,
+                "label": "Method",
+                "multi": true,
+                "name": "method",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view=~\"$view\"}, method)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            }
+        ]
+    },
+    "time": {
+        "from": "now-30m",
+        "to": "now"
+    },
+    "timepicker": {},
+    "timezone": "browser",
+    "title": "Requests / By View",
+    "uid": "backend-requests-by-view",
+    "version": 2,
+    "weekStart": "monday"
+}

infrastructure/grafana/provisioning/dashboards/Celery/by-task.json

@@ -0,0 +1,768 @@
+{
+    "annotations": {
+        "list": [
+            {
+                "builtIn": 1,
+                "datasource": {
+                    "type": "grafana",
+                    "uid": "-- Grafana --"
+                },
+                "enable": true,
+                "hide": true,
+                "iconColor": "rgba(0, 211, 255, 1)",
+                "name": "Annotations & Alerts",
+                "type": "dashboard"
+            }
+        ]
+    },
+    "description": "A dashboard that monitors Celery.",
+    "editable": true,
+    "fiscalYearStartMonth": 0,
+    "graphTooltip": 0,
+    "id": 2,
+    "links": [
+        {
+            "tags": ["celery"],
+            "targetBlank": true,
+            "title": "Celery Dashboards",
+            "type": "dashboards"
+        }
+    ],
+    "liveNow": true,
+    "panels": [
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 0
+            },
+            "id": 1,
+            "panels": [],
+            "title": "Tasks",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "align": "auto",
+                        "cellOptions": {
+                            "type": "auto"
+                        },
+                        "inspect": false
+                    },
+                    "mappings": [],
+                    "noValue": 0,
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "Success Rate"
+                        },
+                        "properties": [
+                            {
+                                "id": "unit",
+                                "value": "percentunit"
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 16,
+                "x": 0,
+                "y": 1
+            },
+            "id": 2,
+            "options": {
+                "cellHeight": "sm",
+                "footer": {
+                    "countRows": false,
+                    "enablePagination": true,
+                    "fields": "",
+                    "reducer": ["sum"],
+                    "show": false
+                },
+                "showHeader": true,
+                "sortBy": [
+                    {
+                        "desc": true,
+                        "displayName": "Succeeded"
+                    }
+                ]
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n/(sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n+sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n) > -1\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_sent_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "D"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_received_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "E"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_rejected_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "F"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_retried_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "G"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_revoked_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "H"
+                }
+            ],
+            "title": "Task Stats",
+            "transformations": [
+                {
+                    "id": "merge"
+                },
+                {
+                    "id": "organize",
+                    "options": {
+                        "excludeByName": {
+                            "Time": true
+                        },
+                        "indexByName": {
+                            "Value #A": 1,
+                            "Value #B": 2,
+                            "Value #C": 3,
+                            "Value #D": 4,
+                            "Value #E": 5,
+                            "Value #F": 6,
+                            "Value #G": 7,
+                            "Value #H": 8,
+                            "name": 0
+                        },
+                        "renameByName": {
+                            "Value #A": "Success Rate",
+                            "Value #B": "Succeeded",
+                            "Value #C": "Failed",
+                            "Value #D": "Sent",
+                            "Value #E": "Received",
+                            "Value #F": "Rejected",
+                            "Value #G": "Retried",
+                            "Value #H": "Revoked",
+                            "name": "Name"
+                        }
+                    }
+                }
+            ],
+            "type": "table"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "align": "auto",
+                        "cellOptions": {
+                            "type": "auto"
+                        },
+                        "inspect": false
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 8,
+                "x": 16,
+                "y": 1
+            },
+            "id": 3,
+            "options": {
+                "cellHeight": "sm",
+                "footer": {
+                    "countRows": false,
+                    "enablePagination": true,
+                    "fields": "",
+                    "reducer": ["sum"],
+                    "show": false
+                },
+                "showHeader": true,
+                "sortBy": [
+                    {
+                        "desc": true,
+                        "displayName": "Value"
+                    }
+                ]
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum (\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  ) by (name, exception) > 0\n)\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "A"
+                }
+            ],
+            "title": "Task Exceptions",
+            "transformations": [
+                {
+                    "id": "organize",
+                    "options": {
+                        "excludeByName": {
+                            "Time": true,
+                            "job": true
+                        },
+                        "indexByName": {
+                            "Value": 2,
+                            "exception": 1,
+                            "name": 0
+                        },
+                        "renameByName": {
+                            "exception": "Exception",
+                            "name": "Task"
+                        }
+                    }
+                }
+            ],
+            "type": "table"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 0,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 9
+            },
+            "id": 4,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Succeeded - {{ name }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Failed - {{ name }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_sent_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Sent - {{ name }}",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_received_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Received - {{ name }}",
+                    "refId": "D"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_retried_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Retried - {{ name }}",
+                    "refId": "E"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_revoked_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Revoked - {{ name }}",
+                    "refId": "F"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_rejected_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Rejected - {{ name }}",
+                    "refId": "G"
+                }
+            ],
+            "title": "Tasks Completed",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 0,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 17
+            },
+            "id": 5,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name, exception) > 0\n",
+                    "legendFormat": "{{ name }}/{{ exception }}",
+                    "refId": "A"
+                }
+            ],
+            "title": "Task Exceptions",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "spanNulls": false
+                    },
+                    "unit": "s"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P50"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "green",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P95"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "yellow",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P99"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "red",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 25
+            },
+            "id": 6,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "v11.1.0",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P50 - {{ name }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P95 - {{ name }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.99,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P99 - {{ name }}",
+                    "refId": "C"
+                }
+            ],
+            "title": "Tasks Runtime",
+            "type": "timeseries"
+        }
+    ],
+    "preload": false,
+    "refresh": "10s",
+    "schemaVersion": 41,
+    "tags": ["celery"],
+    "templating": {
+        "list": [
+            {
+                "label": "Data source",
+                "name": "datasource",
+                "query": "prometheus",
+                "refresh": 1,
+                "type": "datasource"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Namespace",
+                "name": "namespace",
+                "query": "label_values(celery_worker_up{}, namespace)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Job",
+                "name": "job",
+                "query": "label_values(celery_worker_up{namespace=\"$namespace\"}, job)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Queue Name",
+                "name": "queue_name",
+                "query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", name!~\"None\"}, queue_name)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Task",
+                "multi": true,
+                "name": "task",
+                "query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", queue_name=~\"$queue_name\", name!~\"None\"}, name)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            }
+        ]
+    },
+    "time": {
+        "from": "now-24h",
+        "to": "now"
+    },
+    "timepicker": {},
+    "timezone": "browser",
+    "title": "By Task",
+    "uid": "celery-tasks-by-task",
+    "version": 1,
+    "weekStart": "monday"
+}

infrastructure/grafana/provisioning/dashboards/providers.yaml

@@ -1,11 +1,13 @@
 apiVersion: 1
 
 providers:
-  - name: "celery"
+  - name: "default"
     orgId: 1
     folder: ""
     type: file
+    disableDeletion: false
     updateIntervalSeconds: 10
+    allowUiUpdates: false
     options:
       path: /etc/grafana/provisioning/dashboards
       foldersFromFilesStructure: true

infrastructure/grafana/provisioning/datasources/datasources.yaml

@@ -1,10 +1,29 @@
 apiVersion: 1
 
 datasources:
-  - name: Infinity
+  - name: Backend | Infinity
     type: yesoreyeram-infinity-datasource
     access: proxy
     orgId: 1
     uid: infinity
+    url: http://backend:8080
+    basicAuth: false
+    basicAuthUser: ""
+    withCredentials: false
     isDefault: false
     editable: false
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    orgId: 1
+    uid: prometheus
+    url: http://prometheus:9090
+    basicAuth: false
+    basicAuthUser: ""
+    withCredentials: false
+    isDefault: true
+    editable: false
+    jsonData:
+      httpMethod: POST
+      queryTimeout: 10s
+      timeInterval": 10s

infrastructure/loadtest/.env.template

@@ -0,0 +1 @@
+BACKEND_ADDRESS=http://backend:8080

infrastructure/pgadmin/.env.template

@@ -1,2 +1,4 @@
 PGADMIN_DEFAULT_EMAIL=admin@mail.com
-PGADMIN_DEFAULT_PASSWORD_FILE=/run/secrets/pgadmin_password
+PGADMIN_DEFAULT_PASSWORD=password
+PGADMIN_DISABLE_POSTFIX=True
+PGADMIN_REPLACE_SERVERS_ON_STARTUP=True

infrastructure/postgres/.env.template

@@ -1,3 +1,3 @@
 POSTGRES_DB=postgres
 POSTGRES_USER=postgres
-POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
+POSTGRES_PASSWORD=postgres

infrastructure/prometheus/prometheus.yaml

@@ -1,7 +1,7 @@
 # Global config
 global:
-  scrape_interval: 10s
+  scrape_interval: 5s
-  scrape_timeout: 10s
+  scrape_timeout: 5s
   evaluation_interval: 10m
   external_labels:
     environment: local
@@ -10,24 +10,31 @@ global:
 scrape_configs:
   # Prometheus
   - job_name: prometheus
-    scrape_interval: 5s
     static_configs:
       - targets: ["localhost:9090"]
 
   # Postgres
   - job_name: postgres
-    scrape_interval: 10s
     static_configs:
       - targets: ["postgres-exporter:9187"]
 
   # Redis
   - job_name: redis
-    scrape_interval: 10s
     static_configs:
       - targets: ["redis-exporter:9121"]
 
   # Celery
   - job_name: celery
-    scrape_interval: 30s
+    scrape_interval: 15s
     static_configs:
       - targets: ["celery-exporter:9808"]
+
+  # Backend
+  - job_name: backend
+    static_configs:
+      - targets: ["backend:8080"]
+
+  # Caddy
+  - job_name: caddy
+    static_configs:
+      - targets: ["proxy:2019"]

services/backend/.dockerignore

@@ -183,3 +183,6 @@ Dockerfile.staticfiles
 
 # Collected static files
 static
+
+# Profile files
+*.prof

services/backend/.gitignore

@@ -171,3 +171,6 @@ cython_debug/
 
 # Collected static files
 static
+
+# Profile files
+*.prof

services/backend/Dockerfile

@@ -32,11 +32,12 @@ USER app
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
     PYTHONOPTIMIZE=2 \
-    PATH="/opt/venv/bin:$PATH"
+    PATH="/opt/venv/bin:$PATH" \
+    DJANGO_SETTINGS_MODULE=config.settings
 
 EXPOSE 8080
 
 HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --start-interval=2s --retries=3 \
     CMD wget --no-verbose --tries=1 --spider http://127.0.0.1:8080/health?format=json || exit 1
 
-CMD [ "gunicorn", "config.wsgi", "--workers=8", "-b", "0.0.0.0:8080", "--access-logfile", "-", "--error-logfile", "-" ]
+CMD [ "opentelemetry-instrument", "--service_name", "backend-django", "--traces_exporter", "zipkin_json", "gunicorn", "config.wsgi", "--workers=2", "-b", "0.0.0.0:8080", "--access-logfile", "-", "--error-logfile", "-" ]

services/backend/api/v1/ads/views.py

@@ -4,6 +4,7 @@ from uuid import UUID
 from django.http import Http404, HttpRequest
 from django.shortcuts import get_object_or_404
 from ninja import Router
+from silk.profiling.profiler import silk_profile
 
 from api.v1 import schemas as global_schemas
 from api.v1.ads import schemas
@@ -21,6 +22,7 @@ router = Router(tags=["ads"])
         status.NOT_FOUND: global_schemas.NotFoundError,
     },
 )
+@silk_profile("Get Advertisment")
 def get_advertisment(
     request: HttpRequest, client_id: UUID
 ) -> tuple[status, Campaign]:

services/backend/config/settings.py

@@ -4,6 +4,7 @@ import contextlib
 import logging
 from collections.abc import Callable
 from pathlib import Path
+from typing import TYPE_CHECKING
 
 import django_stubs_ext
 import environ
@@ -12,6 +13,10 @@ from health_check.plugins import plugin_dir
 
 from integrations.yandexai.healthcheck import YandexAIHealthCheck
 
+if TYPE_CHECKING:
+    from django.contrib.auth.models import User
+
+
 BASE_DIR = Path(__file__).resolve().parent.parent
 
 env = environ.Env()
@@ -53,7 +58,7 @@ REDIS_URI = env("REDIS_URI", default="redis://localhost:6379")
 
 CACHES = {
     "default": {
-        "BACKEND": "django.core.cache.backends.redis.RedisCache",
+        "BACKEND": "django_prometheus.cache.backends.redis.RedisCache",
         "LOCATION": REDIS_URI,
         "TIMEOUT": None,
         "KEY_PREFIX": "backend",
@@ -80,6 +85,9 @@ CELERY_TASK_TRACK_STARTED = True
 # Database
 
 DB_URI = env.db_url("DJANGO_DB_URI", default="sqlite:///db.sqlite3")
+DB_URI["ENGINE"] = DB_URI["ENGINE"].replace(
+    "django.db.backends", "django_prometheus.db.backends"
+)
 
 DATABASES = {"default": {**DB_URI, "CONN_MAX_AGE": 50}}
 
@@ -284,12 +292,15 @@ INTERNAL_IPS = env(
 )
 
 MIDDLEWARE = [
+    "silk.middleware.SilkyMiddleware",
+    "django_prometheus.middleware.PrometheusBeforeMiddleware",
     "django_guid.middleware.guid_middleware",
     "corsheaders.middleware.CorsMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
     "django.middleware.csrf.CsrfViewMiddleware",
     "django.contrib.auth.middleware.AuthenticationMiddleware",
     "django.contrib.messages.middleware.MessageMiddleware",
+    "django_prometheus.middleware.PrometheusAfterMiddleware",
 ]
 
 SIGNING_BACKEND = "django.core.signing.TimestampSigner"
@@ -442,8 +453,10 @@ INSTALLED_APPS = [
     "corsheaders",
     "django_extensions",
     "django_guid",
+    "django_prometheus",
     "ninja",
     "minio_storage",
+    "silk",
     # Internal apps
     "apps.core",
     "apps.advertiser",
@@ -516,6 +529,13 @@ SECRET_KEY = env("DJANGO_SECRET_KEY", default="very_insecure_key")
 SECRET_KEY_FALLBACKS: list[str] = []
 
 
+# Auth
+
+LOGIN_REDIRECT_URL = "/admin/"
+
+LOGIN_URL = "/admin/"
+
+
 # Sessions
 
 SESSION_CACHE_ALIAS = "default"
@@ -594,3 +614,63 @@ DEBUG_TOOLBAR_CONFIG = {"SHOW_COLLAPSED": True, "UPDATE_ON_FETCH": True}
 if DEBUG and DEBUG_TOOLBAR_ENABLED:
     INSTALLED_APPS.append("debug_toolbar")
     MIDDLEWARE.append("debug_toolbar.middleware.DebugToolbarMiddleware")
+
+
+# Prometheus
+
+PROMETHEUS_LATENCY_BUCKETS = (
+    0.005,
+    0.01,
+    0.025,
+    0.05,
+    0.075,
+    0.1,
+    0.25,
+    0.5,
+    0.75,
+    1.0,
+    2.5,
+    5.0,
+    7.5,
+    10.0,
+    25.0,
+    50.0,
+    75.0,
+    float("inf"),
+)
+
+
+# django-silk
+
+SILKY_PYTHON_PROFILER = True
+
+SILKY_PYTHON_PROFILER_BINARY = True
+
+SILKY_PYTHON_PROFILER_RESULT_PATH = "./profiles"
+
+SILKY_PYTHON_PROFILER_EXTENDED_FILE_NAME = True
+
+SILKY_AUTHENTICATION = True
+
+SILKY_AUTHORISATION = True
+
+
+def is_allowed_to_use_profiling(user: "User") -> bool:
+    return user.is_staff
+
+
+SILKY_PERMISSIONS = is_allowed_to_use_profiling
+
+SILKY_MAX_RECORDED_REQUESTS = 10**3
+
+SILKY_MAX_RECORDED_REQUESTS_CHECK_PERCENT = 10
+
+SILKY_MAX_REQUEST_BODY_SIZE = 128
+
+SILKY_INTERCEPT_PERCENT = 25
+
+SILKY_META = True
+
+SILKY_DYNAMIC_PROFILING = [
+    {"module": "api.v1.ads.views", "function": "get_advertisment"}
+]

services/backend/config/urls.py

@@ -16,6 +16,10 @@ urlpatterns = [
     path("admin/", admin.site.urls),
     # API urls
     path("", include("api.urls")),
+    # Prometheus urls
+    path("", include("django_prometheus.urls")),
+    # Django-silk
+    path("silk/", include("silk.urls", namespace="silk")),
 ]
 
 

services/backend/pyproject.toml

@@ -9,9 +9,29 @@ dependencies = [
  "django-health-check>=3.18.3,<4.0.0",
  "django-minio-storage>=0.5.7,<0.6.0",
  "django-ninja>=1.3.0,<2.0.0",
+ "django-prometheus>=2.4.1,<3.0.0",
+ "django-redis>=6.0.0,<7.0.0",
+ "django-silk[formatting]>=5.4.0,<6.0.0",
  "django-stubs-ext>=5.1.3,<6.0.0",
  "gunicorn>=23.0.0,<24.0.0",
  "httpx>=0.28.1,<0.29.0",
+ "opentelemetry-api>=1.35.0",
+ "opentelemetry-distro>=0.56b0",
+ "opentelemetry-exporter-otlp>=1.35.0",
+ "opentelemetry-exporter-zipkin-proto-http>=1.11.1",
+ "opentelemetry-instrumentation-asyncio>=0.56b0",
+ "opentelemetry-instrumentation-celery>=0.56b0",
+ "opentelemetry-instrumentation-dbapi>=0.56b0",
+ "opentelemetry-instrumentation-django>=0.56b0",
+ "opentelemetry-instrumentation-httpx>=0.56b0",
+ "opentelemetry-instrumentation-psycopg2>=0.56b0",
+ "opentelemetry-instrumentation-requests>=0.56b0",
+ "opentelemetry-instrumentation-sqlite3>=0.56b0",
+ "opentelemetry-instrumentation-threading>=0.56b0",
+ "opentelemetry-instrumentation-urllib>=0.56b0",
+ "opentelemetry-instrumentation-urllib3>=0.56b0",
+ "opentelemetry-instrumentation-wsgi>=0.56b0",
+ "opentelemetry-sdk>=1.35.0",
  "pillow>=11.1.0,<12.0.0",
  "psycopg2-binary>=2.9.10,<3.0.0",
  "pydantic>=2.10.5,<3.0.0",

services/backend/scripts/initdb

@@ -11,3 +11,4 @@ if [ "$DJANGO_CREATE_SUPERUSER" = "True" ]; then
 fi
 
 python manage.py init_cache
+python manage.py silk_clear_request_log

services/loadtest/.dockerignore

@@ -0,0 +1,38 @@
+# If you prefer the allow list template instead of the deny list, see community template:
+# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
+#
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+
+# Test binary, built with `go test -c`
+*.test
+
+# Code coverage profiles and other test artifacts
+*.out
+coverage.*
+*.coverprofile
+profile.cov
+
+# Dependency directories (remove the comment below to include it)
+# vendor/
+
+# Go workspace file
+go.work
+go.work.sum
+
+# env file
+.env
+
+# Editor/IDE
+# .idea/
+# .vscode/
+
+# App binary
+main
+
+# Gitignore
+.gitignore

services/loadtest/.env.template

@@ -0,0 +1,4 @@
+# Change all vars before going to production and remove all comments (!)
+# Below all environment variables and default values
+
+BACKEND_ADDRESS=http://localhost:8080

services/loadtest/.gitignore

@@ -0,0 +1,35 @@
+# If you prefer the allow list template instead of the deny list, see community template:
+# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
+#
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+
+# Test binary, built with `go test -c`
+*.test
+
+# Code coverage profiles and other test artifacts
+*.out
+coverage.*
+*.coverprofile
+profile.cov
+
+# Dependency directories (remove the comment below to include it)
+# vendor/
+
+# Go workspace file
+go.work
+go.work.sum
+
+# env file
+.env
+
+# Editor/IDE
+# .idea/
+# .vscode/
+
+# App binary
+main

services/loadtest/Dockerfile

@@ -0,0 +1,28 @@
+ARG GOARCH=amd64
+
+# Stage 1: Build go binary
+FROM docker.io/golang:1.24-alpine AS build
+
+WORKDIR /build
+
+COPY . .
+
+RUN go mod download
+
+ARG GOARCH
+
+RUN CGO_ENABLED=0 GOARCH=$GOARCH go build -o loadtest
+
+
+# Stage 2: Run go binary
+FROM docker.io/alpine:3.22
+
+WORKDIR /app
+
+COPY --from=build /build/loadtest .
+
+COPY . .
+
+EXPOSE 5001
+
+CMD [ "./loadtest", "--port", "5001" ]

services/loadtest/README.md

@@ -0,0 +1,68 @@
+# AdNova Loadtest
+
+## Prerequisites
+
+Ensure you have the following installed on your system:
+
+- [Go](https://go.dev/) (1.24 recommended)
+- [Docker](https://www.docker.com/) (for containerized setup, latest version recommended)
+
+## Basic setup
+
+### Installation
+
+#### Clone the project
+
+#### Go to the project directory
+
+```bash
+cd AdNova/services/loadtest
+```
+
+#### Customize environment
+
+```bash
+cp .env.template .env
+```
+
+And setup env vars according to your needs.
+
+#### Install dependencies
+
+```bash
+go mod download
+```
+
+#### Running
+
+```bash
+go run main.go
+```
+
+## Containerized setup
+
+### Clone the project
+
+### Go to the project directory
+
+```bash
+cd AdNova/services/loadtest
+```
+
+### Build docker image
+
+```bash
+docker build -t adnova-loadtest .
+```
+
+### Customize environment
+
+Customize environment with `docker run` command, for all environment vars and default values see [.env.template](./.env.template).
+
+### Run docker image
+
+```bash
+docker run -p 5001:5001 --name adnova-loadtest adnova-loadtest
+```
+
+Loadtest will be available on [127.0.0.1:5001](http://127.0.0.1:5001).

services/loadtest/go.mod

@@ -0,0 +1,8 @@
+module loadtest
+
+go 1.24.5
+
+require (
+	github.com/gorilla/mux v1.8.1
+	github.com/gorilla/websocket v1.5.3
+)

services/loadtest/go.sum

@@ -0,0 +1,4 @@
+github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
+github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
+github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
+github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=

services/loadtest/main.go

@@ -0,0 +1,788 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"html/template"
+	"io"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/gorilla/mux"
+	"github.com/gorilla/websocket"
+)
+
+// --- Mock Data Structures ---
+
+type Client struct {
+	ClientID string `json:"client_id"`
+	Login    string `json:"login"`
+	Age      int    `json:"age"`
+	Location string `json:"location"`
+	Gender   string `json:"gender"`
+}
+
+type Advertiser struct {
+	AdvertiserID string `json:"advertiser_id"`
+	Name         string `json:"name"`
+}
+
+type MLScore struct {
+	AdvertiserId string `json:"advertiser_id"`
+	ClientID     string `json:"client_id"`
+	Score        int64  `json:"score"`
+}
+
+type Campaign struct {
+	Targeting struct {
+		Gender   string `json:"gender,omitempty"`
+		AgeFrom  int    `json:"age_from,omitempty"`
+		AgeTo    int    `json:"age_to,omitempty"`
+		Location string `json:"location,omitempty"`
+	} `json:"targeting"`
+	AdTitle           string  `json:"ad_title"`
+	AdText            string  `json:"ad_text"`
+	ImpressionsLimit  int     `json:"impressions_limit"`
+	ClicksLimit       int     `json:"clicks_limit"`
+	CostPerImpression float64 `json:"cost_per_impression"`
+	CostPerClick      float64 `json:"cost_per_click"`
+	StartDate         int64   `json:"start_date"`
+	EndDate           int64   `json:"end_date"`
+}
+
+type CampaignFileEntry struct {
+	AdvertiserID string   `json:"advertiser_id"`
+	CampaignData Campaign `json:"campaign_data"`
+}
+
+// --- Mock data filenames ---
+
+const clientsMockDataFile = "./mocks/bulk_clients.json"
+const advertisersMockDataFile = "./mocks/bulk_advertisers.json"
+const campaignsMockDataFile = "./mocks/campaigns.json"
+const mlscoresMockDataFile = "./mocks/ml_scores.json"
+
+// --- In-Memory Data Store ---
+
+var (
+	loadedClientIDs []string
+	dataMutex       = &sync.RWMutex{}
+	backendAddress  string
+)
+
+// --- Mock Data Loading and Posting Logic ---
+
+func loadInitialMockIDs() {
+	dataMutex.Lock()
+	defer dataMutex.Unlock()
+
+	loadedClientIDs = nil
+
+	log.Println("Loading client IDs from local files...")
+
+	readFile := func(filename string) ([]byte, error) {
+		content, err := os.ReadFile(filename)
+		if err != nil {
+			return nil, fmt.Errorf("error reading %s: %w", filename, err)
+		}
+		return content, nil
+	}
+
+	clientsContent, err := readFile(clientsMockDataFile)
+	if err != nil {
+		log.Printf("Warning: Could not read %s for initial ID load: %v", clientsMockDataFile, err)
+	} else {
+		var tempClients []Client
+		if err := json.Unmarshal(clientsContent, &tempClients); err != nil {
+			log.Printf("Warning: Error unmarshaling %s for initial ID load: %v", clientsMockDataFile, err)
+		} else {
+			for _, c := range tempClients {
+				loadedClientIDs = append(loadedClientIDs, c.ClientID)
+			}
+			log.Printf("Loaded %d client IDs for stress testing.", len(loadedClientIDs))
+		}
+	}
+}
+
+func loadAndPostMocks() error {
+	log.Println("Attempting to load mock data from files and post to external backend:", backendAddress)
+
+	readFile := func(filename string) ([]byte, error) {
+		content, err := os.ReadFile(filename)
+		if err != nil {
+			return nil, fmt.Errorf("error reading %s: %w", filename, err)
+		}
+		return content, nil
+	}
+
+	postJSON := func(url string, data interface{}) error {
+		jsonData, err := json.Marshal(data)
+		if err != nil {
+			return fmt.Errorf("failed to marshal JSON for %s: %w", url, err)
+		}
+		req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
+		if err != nil {
+			return fmt.Errorf("failed to create request for %s: %w", url, err)
+		}
+		req.Header.Set("Content-Type", "application/json")
+
+		client := &http.Client{
+			Timeout: 60 * time.Second,
+		}
+
+		resp, err := client.Do(req)
+		if err != nil {
+			return fmt.Errorf("failed to POST to %s: %w", url, err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode >= 400 {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("POST to %s failed with status %d: %s", url, resp.StatusCode, string(body))
+		}
+		return nil
+	}
+
+	// 1. Load bulk_clients.json
+	clientsContent, err := readFile(clientsMockDataFile)
+	if err != nil {
+		return err
+	}
+	var tempClients []Client
+	if err := json.Unmarshal(clientsContent, &tempClients); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", clientsMockDataFile, err)
+	}
+	if err := postJSON(fmt.Sprintf("%s/clients/bulk", backendAddress), tempClients); err != nil {
+		return fmt.Errorf("error posting bulk clients: %w", err)
+	}
+	log.Printf("Successfully posted %d clients to %s/clients/bulk", len(tempClients), backendAddress)
+
+	// 2. Load bulk_advertisers.json
+	advertisersContent, err := readFile(advertisersMockDataFile)
+	if err != nil {
+		return err
+	}
+	var tempAdvertisers []Advertiser
+	if err := json.Unmarshal(advertisersContent, &tempAdvertisers); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", advertisersMockDataFile, err)
+	}
+	if err := postJSON(fmt.Sprintf("%s/advertisers/bulk", backendAddress), tempAdvertisers); err != nil {
+		return fmt.Errorf("error posting bulk advertisers: %w", err)
+	}
+	log.Printf("Successfully posted %d advertisers to %s/advertisers/bulk", len(tempAdvertisers), backendAddress)
+
+	// 3. Load campaigns.json
+	campaignsContent, err := readFile(campaignsMockDataFile)
+	if err != nil {
+		return err
+	}
+	var campaignDataList []CampaignFileEntry
+	if err := json.Unmarshal(campaignsContent, &campaignDataList); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", campaignsMockDataFile, err)
+	}
+
+	for i, entry := range campaignDataList {
+		if err := postJSON(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, entry.AdvertiserID), entry.CampaignData); err != nil {
+			log.Printf("Warning: Failed to post campaign for advertiser %s (entry %d): %v", entry.AdvertiserID, i, err)
+		}
+	}
+	log.Printf("Attempted to post campaigns for %d advertisers from %s", len(campaignDataList), campaignsMockDataFile)
+
+	// 4. Load ml_scores.json
+	mlScoresContent, err := readFile(mlscoresMockDataFile)
+	if err != nil {
+		return err
+	}
+	var tempMLScores []MLScore
+	if err := json.Unmarshal(mlScoresContent, &tempMLScores); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", mlscoresMockDataFile, err)
+	}
+	for i, score := range tempMLScores {
+		if err := postJSON(fmt.Sprintf("%s/ml-scores", backendAddress), score); err != nil {
+			log.Printf("Warning: Failed to post ML score %d (client %s): %v", i, score.ClientID, err)
+		}
+	}
+	log.Printf("Attempted to post %d ML scores one by one to %s/ml-scores", len(tempMLScores), backendAddress)
+
+	log.Println("Mock data loading and posting process complete.")
+	return nil
+}
+
+// --- WebSocket Hub ---
+
+var upgrader = websocket.Upgrader{
+	ReadBufferSize:  1024,
+	WriteBufferSize: 1024,
+	CheckOrigin:     func(r *http.Request) bool { return true },
+}
+
+type Hub struct {
+	clients    map[*websocket.Conn]bool
+	broadcast  chan []byte
+	register   chan *websocket.Conn
+	unregister chan *websocket.Conn
+	mutex      sync.Mutex
+}
+
+func newHub() *Hub {
+	return &Hub{
+		broadcast:  make(chan []byte),
+		register:   make(chan *websocket.Conn),
+		unregister: make(chan *websocket.Conn),
+		clients:    make(map[*websocket.Conn]bool),
+	}
+}
+
+func (h *Hub) run() {
+	for {
+		select {
+		case client := <-h.register:
+			h.mutex.Lock()
+			h.clients[client] = true
+			h.mutex.Unlock()
+		case client := <-h.unregister:
+			h.mutex.Lock()
+			if _, ok := h.clients[client]; ok {
+				delete(h.clients, client)
+				client.Close()
+			}
+			h.mutex.Unlock()
+		case message := <-h.broadcast:
+			h.mutex.Lock()
+			for client := range h.clients {
+				err := client.WriteMessage(websocket.TextMessage, message)
+				if err != nil {
+					log.Printf("error: %v", err)
+					client.Close()
+					delete(h.clients, client)
+				}
+			}
+			h.mutex.Unlock()
+		}
+	}
+}
+
+var hub = newHub()
+
+// --- Load Generator ---
+
+type TestConfig struct {
+	BackendAddress string `json:"backendAddress"`
+	MaxRPS         int    `json:"maxRps"`
+	LoadProfile    string `json:"loadProfile"`
+	FromRPS        int    `json:"fromRPS"`
+	ToRPS          int    `json:"toRPS"`
+	StepRPS        int    `json:"stepRps"`
+	StepDuration   int    `json:"stepDuration"`
+	OnceCount      int    `json:"onceCount"`
+}
+
+type RequestResult struct {
+	StatusCode int
+	Latency    time.Duration
+	Error      bool
+}
+
+type TestStats struct {
+	RPS         int     `json:"rps"`
+	Latency     float64 `json:"latency"`
+	ErrorRate   float64 `json:"errorRate"`
+	TotalReqs   int64   `json:"totalReqs"`
+	TotalErrors int64   `json:"totalErrors"`
+	IsRunning   bool    `json:"isRunning"`
+}
+
+type TestManager struct {
+	config     TestConfig
+	isRunning  bool
+	cancelFunc context.CancelFunc
+	mutex      sync.Mutex
+	httpClient *http.Client
+}
+
+var testManager = TestManager{
+	httpClient: &http.Client{
+		Timeout: 10 * time.Second,
+		Transport: &http.Transport{
+			MaxIdleConns:        1000,
+			MaxIdleConnsPerHost: 1000,
+			IdleConnTimeout:     90 * time.Second,
+		},
+	},
+}
+
+func (tm *TestManager) startTest(config TestConfig) {
+	tm.mutex.Lock()
+	if tm.isRunning {
+		tm.mutex.Unlock()
+		log.Println("Test already running.")
+		return
+	}
+
+	tm.config = config
+	tm.isRunning = true
+	ctx, cancel := context.WithCancel(context.Background())
+	tm.cancelFunc = cancel
+	tm.mutex.Unlock()
+
+	go tm.runLoadGenerator(ctx)
+}
+
+func (tm *TestManager) stopTest() {
+	tm.mutex.Lock()
+	if tm.isRunning && tm.cancelFunc != nil {
+		tm.cancelFunc()
+		tm.isRunning = false
+	}
+	tm.mutex.Unlock()
+}
+
+func (tm *TestManager) runLoadGenerator(ctx context.Context) {
+	log.Printf("Starting test with config: %+v\n", tm.config)
+
+	results := make(chan RequestResult, 10000)
+	var wg sync.WaitGroup
+
+	var totalReqs, totalErrors int64
+	go func() {
+		ticker := time.NewTicker(1 * time.Second)
+		defer ticker.Stop()
+		var reqsInSecond int
+		var totalLatency time.Duration
+		var errorsInSecond int
+
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case <-ticker.C:
+				avgLatency := 0.0
+				if reqsInSecond > 0 {
+					avgLatency = float64(totalLatency.Milliseconds()) / float64(reqsInSecond)
+				}
+				errorRate := 0.0
+				if reqsInSecond > 0 {
+					errorRate = float64(errorsInSecond) / float64(reqsInSecond) * 100
+				}
+
+				stats := TestStats{
+					RPS:         reqsInSecond,
+					Latency:     avgLatency,
+					ErrorRate:   errorRate,
+					TotalReqs:   totalReqs,
+					TotalErrors: totalErrors,
+					IsRunning:   true,
+				}
+				jsonStats, _ := json.Marshal(stats)
+				hub.broadcast <- jsonStats
+
+				reqsInSecond = 0
+				totalLatency = 0
+				errorsInSecond = 0
+			case res, ok := <-results:
+				if !ok {
+					return
+				}
+				totalReqs++
+				reqsInSecond++
+				totalLatency += res.Latency
+				if res.Error {
+					totalErrors++
+					errorsInSecond++
+				}
+			}
+		}
+	}()
+
+	numWorkers := 1000
+	if tm.config.MaxRPS > 0 && tm.config.MaxRPS < numWorkers {
+		numWorkers = tm.config.MaxRPS
+	}
+
+	jobs := make(chan string, 10000)
+
+	for i := 0; i < numWorkers; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				case url, ok := <-jobs:
+					if !ok {
+						return
+					}
+					start := time.Now()
+					req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
+					resp, err := tm.httpClient.Do(req)
+					latency := time.Since(start)
+
+					if err != nil {
+						results <- RequestResult{StatusCode: 0, Latency: latency, Error: true}
+						continue
+					}
+
+					results <- RequestResult{StatusCode: resp.StatusCode, Latency: latency, Error: resp.StatusCode >= 500}
+					resp.Body.Close()
+				}
+			}
+		}()
+	}
+
+	go func() {
+		defer close(jobs)
+
+		dataMutex.RLock()
+		if len(loadedClientIDs) == 0 {
+			log.Println("No client IDs loaded for stress test.")
+			dataMutex.RUnlock()
+			return
+		}
+		clientIDsForTest := make([]string, len(loadedClientIDs))
+		copy(clientIDsForTest, loadedClientIDs)
+		dataMutex.RUnlock()
+
+		log.Printf("Stress testing with %d client IDs.", len(clientIDsForTest))
+
+		switch tm.config.LoadProfile {
+		case "const":
+			ticker := time.NewTicker(time.Second / time.Duration(tm.config.MaxRPS))
+			defer ticker.Stop()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				case <-ticker.C:
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+				}
+			}
+		case "line":
+			duration := 10 * time.Second
+			startTime := time.Now()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				default:
+					elapsed := time.Since(startTime)
+					if elapsed >= duration {
+						elapsed = duration
+					}
+
+					progress := float64(elapsed) / float64(duration)
+					currentRPS := float64(tm.config.FromRPS) + (float64(tm.config.ToRPS-tm.config.FromRPS) * progress)
+
+					if currentRPS <= 0 {
+						currentRPS = 1
+					}
+
+					sleepDuration := time.Second / time.Duration(currentRPS)
+					time.Sleep(sleepDuration)
+
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+
+					if elapsed >= duration {
+						constSleepDuration := time.Second / time.Duration(tm.config.ToRPS)
+						for {
+							select {
+							case <-ctx.Done():
+								return
+							default:
+								time.Sleep(constSleepDuration)
+								clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+								url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+								jobs <- url
+							}
+						}
+					}
+				}
+			}
+		case "step":
+			currentRPS := tm.config.FromRPS
+			for currentRPS <= tm.config.ToRPS {
+				log.Printf("Step load: %d RPS for %d seconds", currentRPS, tm.config.StepDuration)
+				stepEndTime := time.After(time.Duration(tm.config.StepDuration) * time.Second)
+
+				sleepDuration := time.Second / time.Duration(currentRPS)
+
+			stepLoop:
+				for {
+					select {
+					case <-ctx.Done():
+						return
+					case <-stepEndTime:
+						break stepLoop
+					default:
+						time.Sleep(sleepDuration)
+						clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+						url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+						jobs <- url
+					}
+				}
+				currentRPS += tm.config.StepRPS
+				if tm.config.StepRPS == 0 && currentRPS != tm.config.ToRPS {
+					break
+				}
+			}
+		case "once":
+			for i := 0; i < tm.config.OnceCount; i++ {
+				select {
+				case <-ctx.Done():
+					return
+				default:
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+				}
+			}
+		case "unlimited":
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				default:
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+				}
+			}
+		}
+	}()
+
+	wg.Wait()
+	close(results)
+	time.Sleep(1 * time.Second)
+
+	tm.mutex.Lock()
+	tm.isRunning = false
+	tm.mutex.Unlock()
+
+	finalStats := TestStats{IsRunning: false, TotalReqs: totalReqs, TotalErrors: totalErrors}
+	jsonStats, _ := json.Marshal(finalStats)
+	hub.broadcast <- jsonStats
+	log.Println("Test finished.")
+}
+
+// --- HTTP Handlers ---
+
+func serveWs(w http.ResponseWriter, r *http.Request) {
+	conn, err := upgrader.Upgrade(w, r, nil)
+	if err != nil {
+		log.Println(err)
+		return
+	}
+	hub.register <- conn
+
+	go func() {
+		defer func() {
+			hub.unregister <- conn
+			conn.Close()
+		}()
+		for {
+			if _, _, err := conn.NextReader(); err != nil {
+				break
+			}
+		}
+	}()
+}
+
+func handleLoadMocks(w http.ResponseWriter, r *http.Request) {
+	if err := loadAndPostMocks(); err != nil {
+		http.Error(w, fmt.Sprintf("Failed to load mocks: %v", err), http.StatusInternalServerError)
+		return
+	}
+	w.WriteHeader(http.StatusOK)
+	json.NewEncoder(w).Encode(map[string]string{"status": "Mocks loaded successfully"})
+}
+
+func checkEndpoint(url string) bool {
+	req, err := http.NewRequest("GET", url, nil)
+	if err != nil {
+		log.Printf("Error creating request for %s: %v", url, err)
+		return false
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	req = req.WithContext(ctx)
+
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		log.Printf("Error making request to %s: %v", url, err)
+		return false
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		log.Printf("Check for %s failed with status: %d", url, resp.StatusCode)
+		return false
+	}
+
+	if strings.Contains(url, "/campaigns") {
+		bodyBytes, err := io.ReadAll(resp.Body)
+		if err != nil {
+			log.Printf("Error reading response body from %s: %v", url, err)
+			return false
+		}
+		var campaignsData []Campaign
+		if err := json.Unmarshal(bodyBytes, &campaignsData); err != nil {
+			log.Printf("Error unmarshaling campaign data from %s: %v", url, err)
+			return false
+		}
+		return len(campaignsData) > 0
+	}
+	return true
+}
+
+func handleCheckMocks(w http.ResponseWriter, r *http.Request) {
+	dataMutex.RLock()
+	defer dataMutex.RUnlock()
+
+	status := make(map[string]bool)
+
+	var tempClients []Client
+	clientsContent, err := os.ReadFile(clientsMockDataFile)
+	if err != nil {
+		log.Printf("Warning: Could not read %s for checks: %v", clientsMockDataFile, err)
+	} else {
+		json.Unmarshal(clientsContent, &tempClients)
+	}
+
+	var tempAdvertisers []Advertiser
+	advertisersContent, err := os.ReadFile(advertisersMockDataFile)
+	if err != nil {
+		log.Printf("Warning: Could not read %s for checks: %v", advertisersMockDataFile, err)
+	} else {
+		json.Unmarshal(advertisersContent, &tempAdvertisers)
+	}
+
+	clientsLoaded := false
+	if len(tempClients) >= 3 {
+		firstClient := tempClients[0].ClientID
+		medianClient := tempClients[len(tempClients)/2].ClientID
+		lastClient := tempClients[len(tempClients)-1].ClientID
+
+		clientsLoaded = checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, firstClient)) &&
+			checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, medianClient)) &&
+			checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, lastClient))
+	} else if len(tempClients) > 0 {
+		clientsLoaded = checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, tempClients[0].ClientID))
+	}
+	status["clients"] = clientsLoaded
+
+	advertisersLoaded := false
+	if len(tempAdvertisers) >= 3 {
+		firstAdvertiser := tempAdvertisers[0].AdvertiserID
+		medianAdvertiser := tempAdvertisers[len(tempAdvertisers)/2].AdvertiserID
+		lastAdvertiser := tempAdvertisers[len(tempAdvertisers)-1].AdvertiserID
+
+		advertisersLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, firstAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, medianAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, lastAdvertiser))
+	} else if len(tempAdvertisers) > 0 {
+		advertisersLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, tempAdvertisers[0].AdvertiserID))
+	}
+	status["advertisers"] = advertisersLoaded
+
+	campaignsLoaded := false
+	if len(tempAdvertisers) >= 3 {
+		firstAdvertiser := tempAdvertisers[0].AdvertiserID
+		medianAdvertiser := tempAdvertisers[len(tempAdvertisers)/2].AdvertiserID
+		lastAdvertiser := tempAdvertisers[len(tempAdvertisers)-1].AdvertiserID
+
+		campaignsLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, firstAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, medianAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, lastAdvertiser))
+	} else if len(tempAdvertisers) > 0 {
+		campaignsLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, tempAdvertisers[0].AdvertiserID))
+	}
+	status["campaigns"] = campaignsLoaded
+
+	_, err = os.ReadFile(mlscoresMockDataFile)
+	status["ml_scores"] = err == nil
+	if !status["ml_scores"] {
+		log.Printf("Warning: ML Scores file not found or readable, assuming not loaded for check.")
+	}
+
+	w.Header().Set("Content-Type", "application/json")
+	json.NewEncoder(w).Encode(status)
+}
+
+func handleStartTest(w http.ResponseWriter, r *http.Request) {
+	var config TestConfig
+	if err := json.NewDecoder(r.Body).Decode(&config); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+	config.BackendAddress = backendAddress
+	go testManager.startTest(config)
+	w.WriteHeader(http.StatusOK)
+	json.NewEncoder(w).Encode(map[string]string{"status": "Test started"})
+}
+
+func handleStopTest(w http.ResponseWriter, r *http.Request) {
+	testManager.stopTest()
+	w.WriteHeader(http.StatusOK)
+	json.NewEncoder(w).Encode(map[string]string{"status": "Test stopped"})
+}
+
+func main() {
+	rand.Seed(time.Now().UnixNano())
+
+	port := flag.String("port", "5002", "Port to run the server on")
+	flag.Parse()
+
+	backendAddress = os.Getenv("BACKEND_ADDRESS")
+	if backendAddress == "" {
+		log.Println("BACKEND_ADDRESS environment variable not set. Defaulting to http://localhost:8080")
+		backendAddress = "http://localhost:8080"
+	}
+
+	loadInitialMockIDs()
+
+	go hub.run()
+
+	r := mux.NewRouter()
+
+	r.HandleFunc("/", serveUI)
+	r.HandleFunc("/ws", serveWs)
+	r.HandleFunc("/api/load-mocks", handleLoadMocks).Methods("POST")
+	r.HandleFunc("/api/check-mocks", handleCheckMocks).Methods("GET")
+	r.HandleFunc("/api/start-test", handleStartTest).Methods("POST")
+	r.HandleFunc("/api/stop-test", handleStopTest).Methods("POST")
+
+	addr := ":" + *port
+	log.Printf("Server starting on port %s. Open http://localhost%s\n", *port, addr)
+	if err := http.ListenAndServe(addr, r); err != nil {
+		log.Fatal("ListenAndServe: ", err)
+	}
+}
+
+// --- Embedded HTML UI ---
+
+func serveUI(w http.ResponseWriter, r *http.Request) {
+	w.Header().Set("Content-Type", "text/html; charset=utf-8")
+
+	t, err := template.ParseFiles("static/index.html")
+	if err != nil {
+		http.Error(w, "Could not load UI template", http.StatusInternalServerError)
+		return
+	}
+
+	err = t.Execute(w, nil)
+	if err != nil {
+		http.Error(w, "Failed to render UI", http.StatusInternalServerError)
+	}
+}

services/loadtest/mocks/bulk_advertisers.json

@@ -0,0 +1,122 @@
+[
+    {
+        "advertiser_id": "addf3407-9265-44b8-8e7e-6b015f63c47d",
+        "name": "Smart Inc"
+    },
+    {
+        "advertiser_id": "4d830921-f650-4f2f-869b-db48208dc952",
+        "name": "Advanced Technologies"
+    },
+    {
+        "advertiser_id": "752ddcd3-e944-4eaf-b1de-5e49977354a7",
+        "name": "Global Group"
+    },
+    {
+        "advertiser_id": "e04d35cd-e24b-4bfd-b5fa-6ac1c12ae8a0",
+        "name": "Global Media"
+    },
+    {
+        "advertiser_id": "8b7f547f-bab5-448d-9997-61cbbd6e3f1c",
+        "name": "Modern Systems"
+    },
+    {
+        "advertiser_id": "57dd000f-32fe-4a49-8dca-bfdc84fec7dd",
+        "name": "Advanced Technologies"
+    },
+    {
+        "advertiser_id": "22d28b06-32c5-434e-b05e-0654fa1cac09",
+        "name": "Global Corp"
+    },
+    {
+        "advertiser_id": "7ba54148-4885-48da-97ba-6ecf7b3a300b",
+        "name": "Tech Solutions"
+    },
+    {
+        "advertiser_id": "00ac5657-1a1a-48f1-815a-2ffdb72b82b8",
+        "name": "Elite Group"
+    },
+    {
+        "advertiser_id": "b9d5a8ce-ae23-4a93-946e-4bd85bddef5c",
+        "name": "Modern Marketing"
+    },
+    {
+        "advertiser_id": "3696b94d-6591-4e15-86f3-8c46ee535e27",
+        "name": "Advanced Systems"
+    },
+    {
+        "advertiser_id": "20024aab-2476-4d35-b96f-f47589f05474",
+        "name": "Premier Technologies"
+    },
+    {
+        "advertiser_id": "a68ef7d6-c50f-40d1-b70f-a32d7cdcd036",
+        "name": "Elite Partners"
+    },
+    {
+        "advertiser_id": "960e2896-3f34-43a3-8aa1-d984bb92b96e",
+        "name": "Future Inc"
+    },
+    {
+        "advertiser_id": "80a7f6a9-5939-4d74-9a7e-e455b9fa0fdd",
+        "name": "Modern Marketing"
+    },
+    {
+        "advertiser_id": "6bb59511-360d-49dc-9626-be2f6b9b3a0a",
+        "name": "Future Solutions"
+    },
+    {
+        "advertiser_id": "d7a27dc9-7d8d-4f25-bea4-2528f88238d1",
+        "name": "Advanced Solutions"
+    },
+    {
+        "advertiser_id": "e6cde283-ca4b-456c-b1fb-c2967918b5ef",
+        "name": "Digital Solutions"
+    },
+    {
+        "advertiser_id": "2a51ba38-bed3-43af-bbb4-99093b72e0d1",
+        "name": "Tech Group"
+    },
+    {
+        "advertiser_id": "95a9a488-8fee-4aa6-9c21-458f8f5523ca",
+        "name": "Elite Systems"
+    },
+    {
+        "advertiser_id": "2fb992b8-4441-47f5-9176-1da68526eafb",
+        "name": "Tech Inc"
+    },
+    {
+        "advertiser_id": "f0d160bd-18b4-4c88-917c-06b699abeeec",
+        "name": "Global Inc"
+    },
+    {
+        "advertiser_id": "d8147d8c-2bcd-4ac8-bab5-82e420509a7f",
+        "name": "Smart Inc"
+    },
+    {
+        "advertiser_id": "94209720-2c10-4a55-bc37-c28838d08012",
+        "name": "Advanced Corp"
+    },
+    {
+        "advertiser_id": "33fae4ce-d252-4b34-af2b-c11e770f65b5",
+        "name": "Smart Advertising"
+    },
+    {
+        "advertiser_id": "2c0261ba-bd3c-4b19-a53c-44efa10819e5",
+        "name": "Digital Marketing"
+    },
+    {
+        "advertiser_id": "0892fbb7-fbc6-45e9-b52f-494258a9c8f8",
+        "name": "Smart Group"
+    },
+    {
+        "advertiser_id": "055af498-567a-4a4e-b25f-60a57b9b591c",
+        "name": "Advanced Systems"
+    },
+    {
+        "advertiser_id": "0dfd7246-4cfa-4721-899b-e6dd9c483626",
+        "name": "Premier Advertising"
+    },
+    {
+        "advertiser_id": "32d75bc6-0779-4081-98b3-58273497c561",
+        "name": "Modern Partners"
+    }
+]

services/loadtest/mocks/campaigns.json

@@ -0,0 +1,572 @@
+[
+    {
+        "advertiser_id": "addf3407-9265-44b8-8e7e-6b015f63c47d",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.5,
+            "ad_title": "Amazing Offer",
+            "ad_text": "Get great savings now!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": "MALE",
+                "age_from": 25,
+                "age_to": 40,
+                "location": "Chicago"
+            }
+        }
+    },
+    {
+        "advertiser_id": "4d830921-f650-4f2f-869b-db48208dc952",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.8,
+            "ad_title": "Incredible Deal",
+            "ad_text": "Discover amazing quality!",
+            "start_date": 0,
+            "end_date": 20,
+            "targeting": {
+                "gender": "FEMALE",
+                "age_from": 30,
+                "age_to": 50,
+                "location": "New York"
+            }
+        }
+    },
+    {
+        "advertiser_id": "752ddcd3-e944-4eaf-b1de-5e49977354a7",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.8,
+            "cost_per_click": 2.2,
+            "ad_title": "Best Service",
+            "ad_text": "Buy unique advantages today!",
+            "start_date": 0,
+            "end_date": 22,
+            "targeting": {
+                "gender": "ALL",
+                "age_from": 18,
+                "age_to": 60,
+                "location": "Los Angeles"
+            }
+        }
+    },
+    {
+        "advertiser_id": "e04d35cd-e24b-4bfd-b5fa-6ac1c12ae8a0",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.1,
+            "cost_per_click": 2.9,
+            "ad_title": "Exclusive Opportunity",
+            "ad_text": "Try our new product now!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "8b7f547f-bab5-448d-9997-61cbbd6e3f1c",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.4,
+            "ad_title": "Special Features",
+            "ad_text": "Experience our service today!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "57dd000f-32fe-4a49-8dca-bfdc84fec7dd",
+        "campaign_data": {
+            "impressions_limit": 5,
+            "clicks_limit": 5,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.1,
+            "ad_title": "Limited Time Offer",
+            "ad_text": "Get it before it's gone!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "22d28b06-32c5-434e-b05e-0654fa1cac09",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.7,
+            "cost_per_click": 2.6,
+            "ad_title": "New Arrival",
+            "ad_text": "Check out our latest product!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "7ba54148-4885-48da-97ba-6ecf7b3a300b",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.3,
+            "ad_title": "Discover More",
+            "ad_text": "Find out what we offer!",
+            "start_date": 0,
+            "end_date": 16,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "00ac5657-1a1a-48f1-815a-2ffdb72b82b8",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.4,
+            "cost_per_click": 2.8,
+            "ad_title": "Best Quality",
+            "ad_text": "Experience the best quality!",
+            "start_date": 0,
+            "end_date": 18,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "b9d5a8ce-ae23-4a93-946e-4bd85bddef5c",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.2,
+            "ad_title": "Limited Edition",
+            "ad_text": "Get it while it lasts!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "3696b94d-6591-4e15-86f3-8c46ee535e27",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Exclusive Access",
+            "ad_text": "Join us for exclusive access!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "20024aab-2476-4d35-b96f-f47589f05474",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.8,
+            "cost_per_click": 2.5,
+            "ad_title": "Best Experience",
+            "ad_text": "Experience the best with us!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "a68ef7d6-c50f-40d1-b70f-a32d7cdcd036",
+        "campaign_data": {
+            "impressions_limit": 5,
+            "clicks_limit": 5,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.7,
+            "ad_title": "Incredible Quality",
+            "ad_text": "Get the quality you deserve!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "960e2896-3f34-43a3-8aa1-d984bb92b96e",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.4,
+            "ad_title": "Best Value",
+            "ad_text": "Get the best value for your money!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "80a7f6a9-5939-4d74-9a7e-e455b9fa0fdd",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.6,
+            "ad_title": "Exclusive Offer",
+            "ad_text": "Don't miss out on this offer!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "6bb59511-360d-49dc-9626-be2f6b9b3a0a",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.7,
+            "cost_per_click": 2.3,
+            "ad_title": "Limited Time Deal",
+            "ad_text": "Act fast to get this deal!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "d7a27dc9-7d8d-4f25-bea4-2528f88238d1",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Discover Our Services",
+            "ad_text": "Find out what we can do for you!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "e6cde283-ca4b-456c-b1fb-c2967918b5ef",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.8,
+            "ad_title": "Get Started Today",
+            "ad_text": "Join us and start saving!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "2a51ba38-bed3-43af-bbb4-99093b72e0d1",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.8,
+            "cost_per_click": 2.5,
+            "ad_title": "Best Choice",
+            "ad_text": "Choose the best for your needs!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "95a9a488-8fee-4aa6-9c21-458f8f5523ca",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.2,
+            "ad_title": "Exclusive Access",
+            "ad_text": "Get exclusive access to our services!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "2fb992b8-4441-47f5-9176-1da68526eafb",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.6,
+            "ad_title": "Best Quality",
+            "ad_text": "Experience the best quality!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "f0d160bd-18b4-4c88-917c-06b699abeeec",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.4,
+            "cost_per_click": 2.3,
+            "ad_title": "Incredible Quality",
+            "ad_text": "Get the quality you deserve!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "d8147d8c-2bcd-4ac8-bab5-82e420509a7f",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.8,
+            "ad_title": "Discover Our Services",
+            "ad_text": "Find out what we can do for you!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "94209720-2c10-4a55-bc37-c28838d08012",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Best Value",
+            "ad_text": "Get the best value for your money!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "33fae4ce-d252-4b34-af2b-c11e770f65b5",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.4,
+            "ad_title": "Exclusive Offer",
+            "ad_text": "Don't miss out on this offer!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "2c0261ba-bd3c-4b19-a53c-44efa10819e5",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.6,
+            "ad_title": "Best Choice",
+            "ad_text": "Choose the best for your needs!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "0892fbb7-fbc6-45e9-b52f-494258a9c8f8",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.3,
+            "ad_title": "Incredible Quality",
+            "ad_text": "Get the quality you deserve!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "055af498-567a-4a4e-b25f-60a57b9b591c",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.8,
+            "ad_title": "Get Started Today",
+            "ad_text": "Join us and start saving!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "0dfd7246-4cfa-4721-899b-e6dd9c483626",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Best Experience",
+            "ad_text": "Experience the best with us!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "32d75bc6-0779-4081-98b3-58273497c561",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.2,
+            "ad_title": "Discover Our Services",
+            "ad_text": "Find out what we can do for you!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    }
+]

services/loadtest/static/index.html

@@ -0,0 +1,459 @@
+<!DOCTYPE html>
+<html lang="en" class="dark">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>AdNova Loadtest</title>
+    <script src="https://cdn.tailwindcss.com"></script>
+    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
+    <style>
+        body {
+            font-family: 'Inter', sans-serif;
+        }
+
+        @import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');
+
+        .chart-container {
+            position: relative;
+            height: 250px;
+            width: 100%;
+        }
+
+        .status-dot {
+            height: 10px;
+            width: 10px;
+            border-radius: 50%;
+            display: inline-block;
+        }
+
+        .tooltip {
+            visibility: hidden;
+        }
+
+        .has-tooltip:hover .tooltip {
+            visibility: visible;
+        }
+    </style>
+</head>
+
+<body class="bg-slate-900 text-gray-300">
+    <div class="container mx-auto p-4 md:p-8">
+        <header class="mb-8">
+            <h1 class="text-4xl font-bold text-white tracking-tight">AdNova Loadtest</h1>
+        </header>
+
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-8">
+            <!-- Left Column: Controls -->
+            <div class="lg:col-span-1 flex flex-col gap-6">
+
+                <!-- Mock Data Management -->
+                <div class="bg-slate-800 rounded-lg p-6 shadow-lg">
+                    <h2 class="text-xl font-semibold text-white mb-4">1. Data Setup</h2>
+                    <button id="loadMocksBtn"
+                        class="w-full bg-indigo-600 hover:bg-indigo-500 text-white font-bold py-2 px-4 rounded-lg transition-colors duration-300 flex items-center justify-center">
+                        Load Mock Data
+                    </button>
+                    <div id="mockStatus" class="mt-4 space-y-2 text-sm">
+                        <div class="flex items-center justify-between"><span class="text-slate-300">Clients:</span><span
+                                id="clientsStatus" class="font-mono">Checking...</span></div>
+                        <div class="flex items-center justify-between"><span
+                                class="text-slate-300">Advertisers:</span><span id="advertisersStatus"
+                                class="font-mono">Checking...</span></div>
+                        <div class="flex items-center justify-between"><span
+                                class="text-slate-300">Campaigns:</span><span id="campaignsStatus"
+                                class="font-mono">Checking...</span></div>
+                        <div class="flex items-center justify-between"><span class="text-slate-300">ML
+                                Scores:</span><span id="mlScoresStatus" class="font-mono">Checking...</span></div>
+                    </div>
+                </div>
+
+                <!-- Test Configuration -->
+                <div id="config-panel" class="bg-slate-800 rounded-lg p-6 shadow-lg">
+                    <h2 class="text-xl font-semibold text-white mb-4">2. Test Configuration</h2>
+                    <div class="space-y-4">
+                        <div>
+                            <label for="loadProfile" class="block text-sm font-medium text-slate-300">Load
+                                Profile</label>
+                            <select id="loadProfile"
+                                class="mt-1 block w-full bg-slate-700 border border-slate-600 rounded-md shadow-sm py-2 px-3 text-white focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm">
+                                <option value="const">Constant</option>
+                                <option value="line">Line</option>
+                                <option value="step">Step</option>
+                                <option value="once">Once</option>
+                                <option value="unlimited">Unlimited</option>
+                            </select>
+                        </div>
+
+                        <!-- Profile Specific Options -->
+                        <div id="const-options">
+                            <label for="maxRps" class="block text-sm font-medium text-slate-300">Requests Per Second
+                                (RPS)</label>
+                            <input type="number" id="maxRps" value="100"
+                                class="mt-1 block w-full bg-slate-700 border border-slate-600 rounded-md shadow-sm py-2 px-3 text-white focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm">
+                        </div>
+                        <div id="line-options" class="hidden space-y-2">
+                            <input type="number" id="fromRps" placeholder="From RPS (e.g., 10)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="toRps" placeholder="To RPS (e.g., 100)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                        </div>
+                        <div id="step-options" class="hidden space-y-2">
+                            <input type="number" id="stepFromRps" placeholder="From RPS"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="stepToRps" placeholder="To RPS"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="stepRps" placeholder="Step Size (RPS)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="stepDuration" placeholder="Step Duration (sec)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                        </div>
+                        <div id="once-options" class="hidden">
+                            <input type="number" id="onceCount" placeholder="Number of Requests"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                        </div>
+                        <div id="unlimited-options"
+                            class="hidden p-2 bg-yellow-900/50 rounded-md text-yellow-300 text-xs">
+                            Warning: This profile sends requests as fast as possible and may exhaust system resources.
+                        </div>
+
+                    </div>
+                </div>
+
+                <!-- Start/Stop Controls -->
+                <div class="bg-slate-800 rounded-lg p-6 shadow-lg">
+                    <h2 class="text-xl font-semibold text-white mb-4">3. Execute Test</h2>
+                    <button id="startStopBtn"
+                        class="w-full bg-green-600 hover:bg-green-500 text-white font-bold py-3 px-4 rounded-lg transition-colors duration-300 text-lg flex items-center justify-center">
+                        Start Test
+                    </button>
+                </div>
+            </div>
+
+            <!-- Right Column: Stats & Charts -->
+            <div class="lg:col-span-2 flex flex-col gap-6">
+                <!-- Summary Stats -->
+                <div class="grid grid-cols-1 md:grid-cols-3 gap-4">
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">RPS</h3>
+                        <p id="rpsStat" class="text-3xl font-semibold text-white">0</p>
+                    </div>
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Avg Latency (ms)</h3>
+                        <p id="latencyStat" class="text-3xl font-semibold text-white">0.00</p>
+                    </div>
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Error Rate</h3>
+                        <p id="errorRateStat" class="text-3xl font-semibold text-white">0.00%</p>
+                    </div>
+                </div>
+                <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Total Requests</h3>
+                        <p id="totalReqsStat" class="text-2xl font-semibold text-white">0</p>
+                    </div>
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Total Errors</h3>
+                        <p id="totalErrorsStat" class="text-2xl font-semibold text-white">0</p>
+                    </div>
+                </div>
+
+                <!-- Charts -->
+                <div class="bg-slate-800 p-4 rounded-lg shadow-lg">
+                    <h3 class="font-semibold text-white mb-2">Requests Per Second (RPS)</h3>
+                    <div class="chart-container"><canvas id="rpsChart"></canvas></div>
+                </div>
+                <div class="bg-slate-800 p-4 rounded-lg shadow-lg">
+                    <h3 class="font-semibold text-white mb-2">Average Latency (ms)</h3>
+                    <div class="chart-container"><canvas id="latencyChart"></canvas></div>
+                </div>
+                <div class="bg-slate-800 p-4 rounded-lg shadow-lg">
+                    <h3 class="font-semibold text-white mb-2">Error Rate (%)</h3>
+                    <div class="chart-container"><canvas id="errorRateChart"></canvas></div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+    <script>
+        const loadProfileSelect = document.getElementById( 'loadProfile' )
+        const startStopBtn = document.getElementById( 'startStopBtn' )
+        const loadMocksBtn = document.getElementById( 'loadMocksBtn' )
+        const configPanel = document.getElementById( 'config-panel' )
+
+        const rpsStat = document.getElementById( 'rpsStat' )
+        const latencyStat = document.getElementById( 'latencyStat' )
+        const errorRateStat = document.getElementById( 'errorRateStat' )
+        const totalReqsStat = document.getElementById( 'totalReqsStat' )
+        const totalErrorsStat = document.getElementById( 'totalErrorsStat' )
+
+        let ws
+        let rpsChart, latencyChart, errorRateChart
+        let isRunning = false
+        const MAX_DATA_POINTS = 60
+
+        function createChart ( ctx, label, color )
+        {
+            return new Chart( ctx, {
+                type: 'line',
+                data: {
+                    labels: [],
+                    datasets: [ {
+                        label: label,
+                        data: [],
+                        borderColor: color,
+                        backgroundColor: color + '33',
+                        borderWidth: 2,
+                        pointRadius: 0,
+                        tension: 0.4,
+                        fill: true,
+                    } ]
+                },
+                options: {
+                    responsive: true,
+                    maintainAspectRatio: false,
+                    scales: {
+                        x: {
+                            ticks: { color: '#94a3b8' },
+                            grid: { color: '#334155' }
+                        },
+                        y: {
+                            beginAtZero: true,
+                            ticks: { color: '#94a3b8' },
+                            grid: { color: '#334155' }
+                        }
+                    },
+                    plugins: {
+                        legend: { display: false }
+                    }
+                }
+            } )
+        }
+
+        function updateChart ( chart, label, newData )
+        {
+            chart.data.labels.push( label )
+            chart.data.datasets[ 0 ].data.push( newData )
+            if ( chart.data.labels.length > MAX_DATA_POINTS )
+            {
+                chart.data.labels.shift()
+                chart.data.datasets[ 0 ].data.shift()
+            }
+            chart.update( 'none' )
+        }
+
+        function resetCharts ()
+        {
+            const charts = [ rpsChart, latencyChart, errorRateChart ]
+            charts.forEach( chart =>
+            {
+                if ( chart )
+                {
+                    chart.data.labels = []
+                    chart.data.datasets[ 0 ].data = []
+                    chart.update()
+                }
+            } )
+            rpsStat.textContent = '0'
+            latencyStat.textContent = '0.00'
+            errorRateStat.textContent = '0.00%'
+            totalReqsStat.textContent = '0'
+            totalErrorsStat.textContent = '0'
+        }
+
+        function setupWebSocket ()
+        {
+            const protocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://'
+            const wsUrl = protocol + window.location.host + '/ws'
+            ws = new WebSocket( wsUrl )
+
+            ws.onopen = () => console.log( 'WebSocket connected' )
+            ws.onclose = () => console.log( 'WebSocket disconnected' )
+            ws.onerror = ( error ) => console.error( 'WebSocket error:', error )
+
+            ws.onmessage = ( event ) =>
+            {
+                const data = JSON.parse( event.data )
+
+                if ( typeof data.isRunning !== 'undefined' )
+                {
+                    if ( data.isRunning )
+                    {
+                        isRunning = true
+                        startStopBtn.textContent = 'Stop Test'
+                        startStopBtn.classList.remove( 'bg-green-600', 'hover:bg-green-500' )
+                        startStopBtn.classList.add( 'bg-red-600', 'hover:bg-red-500' )
+                        configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = true )
+                    }
+                }
+
+                if ( data.isRunning )
+                {
+                    const now = new Date().toLocaleTimeString()
+                    rpsStat.textContent = data.rps
+                    latencyStat.textContent = data.latency.toFixed( 2 )
+                    errorRateStat.textContent = data.errorRate.toFixed( 2 ) + '%'
+                    totalReqsStat.textContent = data.totalReqs
+                    totalErrorsStat.textContent = data.totalErrors
+
+                    updateChart( rpsChart, now, data.rps )
+                    updateChart( latencyChart, now, data.latency )
+                    updateChart( errorRateChart, now, data.errorRate )
+                } else
+                {
+                    totalReqsStat.textContent = data.totalReqs
+                    totalErrorsStat.textContent = data.totalErrors
+                }
+            }
+        }
+
+        function setRunningState ()
+        {
+            isRunning = true
+            startStopBtn.textContent = 'Stop Test'
+            startStopBtn.classList.remove( 'bg-green-600', 'hover:bg-green-500' )
+            startStopBtn.classList.add( 'bg-red-600', 'hover:bg-red-500' )
+            configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = true )
+            resetCharts()
+        }
+
+        function setStoppedState ()
+        {
+            isRunning = false
+            startStopBtn.textContent = 'Start Test'
+            startStopBtn.classList.remove( 'bg-red-600', 'hover:bg-red-500' )
+            startStopBtn.classList.add( 'bg-green-600', 'hover:bg-green-500' )
+            configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = false )
+        }
+
+        startStopBtn.addEventListener( 'click', () =>
+        {
+            if ( isRunning )
+            {
+                fetch( '/api/stop-test', { method: 'POST' } )
+                    .then( () => setStoppedState() )
+            } else
+            {
+                const config = {
+                    loadProfile: loadProfileSelect.value,
+                    maxRps: parseInt( document.getElementById( 'maxRps' ).value ) || 100,
+                    fromRps: parseInt( document.getElementById( 'fromRps' ).value ) || 0,
+                    toRps: parseInt( document.getElementById( 'toRps' ).value ) || 0,
+                    stepFromRps: parseInt( document.getElementById( 'stepFromRps' ).value ) || 0,
+                    stepToRps: parseInt( document.getElementById( 'stepToRps' ).value ) || 0,
+                    stepRps: parseInt( document.getElementById( 'stepRps' ).value ) || 0,
+                    stepDuration: parseInt( document.getElementById( 'stepDuration' ).value ) || 0,
+                    onceCount: parseInt( document.getElementById( 'onceCount' ).value ) || 0,
+                }
+
+                if ( config.loadProfile === 'step' )
+                {
+                    config.fromRps = config.stepFromRps
+                    config.toRps = config.stepToRps
+                }
+
+                fetch( '/api/start-test', {
+                    method: 'POST',
+                    headers: { 'Content-Type': 'application/json' },
+                    body: JSON.stringify( config )
+                } ).then( res =>
+                {
+                    if ( res.ok ) setRunningState()
+                    else alert( "Failed to start test. Check console for details." )
+                } ).catch( err =>
+                {
+                    console.error( "Error starting test:", err )
+                    alert( "Failed to start test. Network error or server unreachable." )
+                } )
+            }
+        } )
+
+        function updateMockStatus ( status )
+        {
+            const statuses = {
+                clients: document.getElementById( 'clientsStatus' ),
+                advertisers: document.getElementById( 'advertisersStatus' ),
+                campaigns: document.getElementById( 'campaignsStatus' ),
+                ml_scores: document.getElementById( 'mlScoresStatus' ),
+            }
+
+            for ( const key in status )
+            {
+                const el = statuses[ key ]
+                if ( el )
+                {
+                    if ( status[ key ] )
+                    {
+                        el.innerHTML = '<span class="status-dot bg-green-500 mr-2"></span>Loaded'
+                        el.classList.remove( 'text-red-400' )
+                        el.classList.add( 'text-green-400' )
+                    } else
+                    {
+                        el.innerHTML = '<span class="status-dot bg-red-500 mr-2"></span>Not Loaded'
+                        el.classList.remove( 'text-green-400' )
+                        el.classList.add( 'text-red-400' )
+                    }
+                }
+            }
+        }
+
+        function checkMocks ()
+        {
+            fetch( '/api/check-mocks' )
+                .then( res => res.json() )
+                .then( data => updateMockStatus( data ) )
+                .catch( err => console.error( "Failed to check mocks:", err ) )
+        }
+
+        loadMocksBtn.addEventListener( 'click', () =>
+        {
+            loadMocksBtn.textContent = 'Loading...'
+            loadMocksBtn.disabled = true
+
+            fetch( '/api/load-mocks', {
+                method: 'POST',
+                headers: { 'Content-Type': 'application/json' },
+                body: JSON.stringify( {} )
+            } )
+                .then( res =>
+                {
+                    if ( !res.ok ) alert( 'Failed to load mocks. Check console for details.' )
+                    return res.json()
+                } )
+                .then( () =>
+                {
+                    checkMocks()
+                } )
+                .finally( () =>
+                {
+                    loadMocksBtn.textContent = 'Load Mock Data'
+                    loadMocksBtn.disabled = false
+                } )
+        } )
+
+        loadProfileSelect.addEventListener( 'change', ( e ) =>
+        {
+            document.getElementById( 'const-options' ).classList.add( 'hidden' )
+            document.getElementById( 'line-options' ).classList.add( 'hidden' )
+            document.getElementById( 'step-options' ).classList.add( 'hidden' )
+            document.getElementById( 'once-options' ).classList.add( 'hidden' )
+            document.getElementById( 'unlimited-options' ).classList.add( 'hidden' )
+
+            document.getElementById( e.target.value + '-options' ).classList.remove( 'hidden' )
+        } )
+
+
+        window.onload = () =>
+        {
+            rpsChart = createChart( document.getElementById( 'rpsChart' ).getContext( '2d' ), 'RPS', '#6366f1' )
+            latencyChart = createChart( document.getElementById( 'latencyChart' ).getContext( '2d' ), 'Latency', '#34d399' )
+            errorRateChart = createChart( document.getElementById( 'errorRateChart' ).getContext( '2d' ), 'Error Rate', '#f87171' )
+
+            setupWebSocket()
+            checkMocks()
+            loadProfileSelect.dispatchEvent( new Event( 'change' ) )
+        };
+    </script>
+</body>
+
+</html>

tests/e2e/.env.template

@@ -1,3 +1,3 @@
 # Below all environment variables and default values
 
-BACKEND_BASE_URL=http://127.0.0.1:8080
+BACKEND_BASE_URL=http://127.0.0.1:13240

tests/e2e/README.md

@@ -11,7 +11,7 @@ Ensure you have the following installed on your system:
 
 ## Warning
 
-Please note that containers will use ports from 13241 to 13246 and 8080, so there is must be no listeners on this ports range.
+Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
 
 ## Setup
 

tests/e2e/conftest.py

@@ -9,7 +9,7 @@ from dotenv import load_dotenv
 
 load_dotenv()
 
-BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:8080")
+BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:13240")
 
 
 @pytest.fixture(scope="session", autouse=True)
@@ -20,6 +20,16 @@ def docker_compose() -> Generator[None]:
         args=[
             "docker",
             "compose",
+            "--profile",
+            "loadtest",
+            "--profile",
+            "minio",
+            "--profile",
+            "observability",
+            "--profile",
+            "proxy",
+            "--profile",
+            "telegram_bot",
             "down",
         ],
         check=True,
@@ -31,6 +41,8 @@ def docker_compose() -> Generator[None]:
             "compose",
             "--project-name",
             "adnova-testing",
+            "--profile",
+            "minio",
             "up",
             "-d",
             "--build",
@@ -50,6 +62,8 @@ def docker_compose() -> Generator[None]:
             "compose",
             "--project-name",
             "adnova-testing",
+            "--profile",
+            "minio",
             "down",
             "-v",
         ],