fix(grafana): fixed false-positive NoData state

btw, they are incorrect

README.md

@@ -32,7 +32,7 @@ Table Report:
 
 #### Warning
 
-Please note that containers will use ports from 13241 to 13245 and 8080, so there is must be no listeners on this ports range.
+Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
 
 #### Configure
 
@@ -66,7 +66,7 @@ docker compose up -d --build
 
 #### Structure
 
-- **backend**: [127.0.0.1:8080](http://127.0.0.1:8080) -> `8080`
+- **backend**: [127.0.0.1:13240](http://127.0.0.1:13240) -> `8080`
   - Depends on: `postgres`, `redis`, `minio`, `backend-initdb`
 - **backend-initdb**
   - Depends on: `postgres`, `redis`, `minio`
@@ -109,7 +109,7 @@ aiogram is a modern and fully asynchronous framework for Telegram bot developmen
 
 ### [Redis](https://redis.io/)
 
-Redis is an in-memory data structure store often used as a database, cache, and message broker. It supports various data structures and offers high performance for read and write operations, making it suitable for caching and real-time analytics. Very popular and has big community for today. In project used as fsm for aiogram (to avoid data loss on restart), caches (current_date, mlscores, clicks, views) for backend and as broker for Celery.
+Redis is an in-memory data structure store often used as a database, cache, and message broker. It supports various data structures and offers high performance for read and write operations, making it suitable for caching and real-time analytics. Very popular and has big community for today. In project used as fsm for aiogram (to avoid data loss on restart), caches (mlscores, clicks, views) for backend and as broker for Celery.
 
 ### [Postgres](https://www.postgresql.org/)
 
@@ -139,7 +139,7 @@ You may say: "For what we need a lot of complex technologies for now". I have an
 
 ### Restful API
 
-API Base endpoint when deployed with default docker compose: [127.0.0.1:8080](http://127.0.0.1:8080), also see [docs](#openapi-docs).
+API Base endpoint when deployed with default docker compose: [127.0.0.1:13240](http://127.0.0.1:13240), also see [docs](#openapi-docs).
 
 ### Admin panel
 
@@ -152,9 +152,13 @@ Link: [t.me/adnova_bot](https://t.me/adnova_bot)
 Basic commands:
 
 `/start` - Start the bot and authenticate as advertiser
+
 `/help` - Get list of all commands
+
 `/campaigns` - Manage advertiser campaigns (only after authentication)
+
 `/statistics` - See advertiser overall statistics (only after authentication)
+
 `/logout` - Logout of current advertiser account (only after authentication)
 
 See [this](#telegram-bot-1).
@@ -234,9 +238,9 @@ Moderation implemented via report system. Client goes to `/report` ([see OpenAPI
 Also admin user (whose credentials specified lower) can add new staff members and even create a specified group for them (this is built-in django capabilities).
 Report has four states: Sent, Under review, Took action and Skipped. Admin panel has filtration by states and by flagged by llm status.
 
-Admin panel when deployed with docker compose (by default): [localhost:8080/admin/](http://localhost:8080/admin/)
+Admin panel when deployed with docker compose (by default): [localhost:13240/admin/](http://localhost:13240/admin/)
 
-Reports list when deployed with docker compose (requires authentication): [localhost:8080/admin/campaign/campaignreport/](http://localhost:8080/admin/campaign/campaignreport/)
+Reports list when deployed with docker compose (requires authentication): [localhost:13240/admin/campaign/campaignreport/](http://localhost:13240/admin/campaign/campaignreport/)
 
 Default username: `admin`
 
@@ -268,7 +272,7 @@ Default login: `admin`
 
 Default password: `proooooood`
 
-Analytics dashboard when deployed with default docker compose: [localhost:13243/d/adnova-statistics/statistics](http://localhost:13243/d/adnova-statisticss/statistics). You can enter advertiser id and get detailed advertiser statistics and also detailed statistics for each advertiser's campaign.
+Analytics dashboard when deployed with default docker compose: [localhost:13243/d/adnova-advertiser-statistics](http://localhost:13243/d/adnova-advertiser-statistics/statistics). You can enter advertiser id and get detailed advertiser statistics and also detailed statistics for each advertiser's campaign.
 
 Demonstration:
 
@@ -276,11 +280,11 @@ Demonstration:
 
 ### OpenAPI docs
 
-When deployed with default docker compose: [localhost:8080/docs](http://localhost:8080/docs)
+When deployed with default docker compose: [localhost:13240/docs](http://localhost:13240/docs)
 
 ### Healthcheck endpoint
 
-When deployed with default docker compose: [localhost:8080/health](http://localhost:8080/health)
+When deployed with default docker compose: [localhost:13240/health](http://localhost:13240/health)
 
 Lets developers easily understand and identify problem and users check services health.
 

compose.yaml

@@ -5,6 +5,9 @@ services:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     depends_on:
       backend-initdb:
         restart: false
@@ -18,10 +21,6 @@ services:
         restart: false
         condition: service_healthy
         required: true
-      minio:
-        restart: false
-        condition: service_healthy
-        required: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
@@ -30,15 +29,20 @@ services:
     ports:
       - name: web
         target: 8080
-        published: 8080
+        published: 13240
         host_ip: 127.0.0.1
         protocol: tcp
+        app_protocol: http
     restart: unless-stopped
+    shm_size: 4mb
 
   backend-initdb:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     command: ./scripts/initdb
     depends_on:
       postgres:
@@ -49,27 +53,27 @@ services:
         restart: false
         condition: service_healthy
         required: true
-      minio:
-        restart: false
-        condition: service_healthy
-        required: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
       - path: ./infrastructure/backend/.env
         required: false
+    shm_size: 4mb
 
   backend-staticfiles:
     build:
       context: ./services/backend
       dockerfile: Dockerfile.staticfiles
+      tags:
+        - adnova-backend-staticfiles:latest
+      pull: true
     env_file:
       - path: ./infrastructure/backend/.env.template
         required: true
       - path: ./infrastructure/backend/.env
         required: false
     healthcheck:
-      test: ["CMD", "service", "nginx", "status", "||", " exit 1"]
+      test: ["CMD-SHELL", "nginx", "-t", "||", "exit 1"]
       interval: 1m30s
       timeout: 5s
       start_period: 5s
@@ -81,12 +85,17 @@ services:
         published: 13241
         host_ip: 127.0.0.1
         protocol: tcp
+        app_protocol: http
     restart: unless-stopped
+    shm_size: 4mb
 
   backend-celery-worker:
     build:
       context: ./services/backend
       dockerfile: Dockerfile
+      tags:
+        - adnova-backend:latest
+      pull: true
     command: celery -A config worker -l INFO
     depends_on:
       redis:
@@ -106,9 +115,11 @@ services:
       start_period: 10s
       start_interval: 2s
     restart: unless-stopped
+    shm_size: 4mb
 
   celery-exporter:
-    image: docker.io/danihodovic/celery-exporter:0.11.1
+    image: docker.io/danihodovic/celery-exporter:0.12.2
+    command: --retry-interval=5
     depends_on:
       redis:
         restart: false
@@ -119,7 +130,10 @@ services:
         required: true
       - path: ./infrastructure/celery-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
+    shm_size: 4mb
 
   telegram_bot:
     build:
@@ -127,6 +141,7 @@ services:
       dockerfile: Dockerfile
       tags:
         - adnova-telegram_bot:latest
+      pull: true
     depends_on:
       backend:
         restart: false
@@ -145,7 +160,10 @@ services:
         required: true
       - path: ./infrastructure/telegram_bot/.env
         required: false
+    profiles:
+      - telegram_bot
     restart: unless-stopped
+    shm_size: 4mb
 
   redis:
     image: docker.io/redis:7-alpine3.21
@@ -171,9 +189,10 @@ services:
       - type: volume
         source: redis_data
         target: /data
+        read_only: false
 
   redis-exporter:
-    image: docker.io/oliver006/redis_exporter:v1.67.0-alpine
+    image: docker.io/oliver006/redis_exporter:v1.74.0-alpine
     depends_on:
       redis:
         restart: false
@@ -184,11 +203,13 @@ services:
         required: true
       - path: ./infrastructure/redis-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
 
   postgres:
-    image: docker.io/postgres:17-alpine3.21
+    image: docker.io/postgres:17-alpine3.22
     configs:
       - source: postgres_config
         target: /etc/postgresql/postgresql.conf
@@ -198,7 +219,7 @@ services:
       - path: ./infrastructure/postgres/.env
         required: false
     healthcheck:
-      test: ["CMD", "pg_isready"]
+      test: ["CMD", "pg_isready", "-U", "postgres"]
       interval: 1m30s
       timeout: 5s
       start_period: 5s
@@ -206,17 +227,15 @@ services:
       retries: 5
     oom_kill_disable: true
     restart: unless-stopped
-    secrets:
-      - source: postgres_password
-        target: /run/secrets/postgres_password
     shm_size: 128mb
     volumes:
       - type: volume
         source: postgres_data
         target: /var/lib/postgresql/data
+        read_only: false
 
   postgres-exporter:
-    image: quay.io/prometheuscommunity/postgres-exporter:v0.16.0
+    image: quay.io/prometheuscommunity/postgres-exporter:v0.17.1
     depends_on:
       postgres:
         restart: false
@@ -227,13 +246,15 @@ services:
         required: true
       - path: ./infrastructure/postgres-exporter/.env
         required: false
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
 
   pgadmin:
-    image: docker.io/dpage/pgadmin4:9
+    image: docker.io/dpage/pgadmin4:9.6
     configs:
-      - source: pgadmin_servers
+      - source: pgadmin_servers_config
         target: /pgadmin4/servers.json
     depends_on:
       postgres:
@@ -258,22 +279,23 @@ services:
         published: 13242
         host_ip: 127.0.0.1
         protocol: tcp
+        app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
-    secrets:
-      - source: pgadmin_password
-        target: /run/secrets/pgadmin_password
     shm_size: 4mb
     volumes:
       - type: volume
         source: pgadmin_data
         target: /var/lib/pgadmin
+        read_only: false
 
   grafana:
-    image: docker.io/grafana/grafana-oss:11.5.0
+    image: docker.io/grafana/grafana-oss:12.0.2
+    entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
     configs:
       - source: grafana_config
         target: /usr/share/grafana/conf/defaults.ini
-    entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
     healthcheck:
       test: ["CMD", "wget", "-O", "-", "http://localhost:3000/api/health"]
       interval: 1m30s
@@ -287,23 +309,28 @@ services:
         published: 13243
         host_ip: 127.0.0.1
         protocol: tcp
+        app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
     volumes:
       - type: volume
         source: grafana_data
         target: /var/lib/grafana
+        read_only: false
       - type: bind
         source: ./infrastructure/grafana/provisioning
         target: /etc/grafana/provisioning
+        read_only: true
       - type: bind
         source: ./infrastructure/grafana/scripts
         target: /etc/grafana/scripts
+        read_only: true
 
   prometheus:
-    image: docker.io/prom/prometheus:v3.1.0
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yaml"
+    image: docker.io/prom/prometheus:v3.5.0
+    command: --config.file=/etc/prometheus/prometheus.yaml
     configs:
       - source: prometheus_config
         target: /etc/prometheus/prometheus.yaml
@@ -320,16 +347,20 @@ services:
         published: 13244
         host_ip: 127.0.0.1
         protocol: tcp
+        app_protocol: http
+    profiles:
+      - observability
     restart: unless-stopped
     shm_size: 4mb
     volumes:
       - type: volume
         source: prometheus_data
         target: /prometheus
+        read_only: false
 
   minio:
+    image: docker.io/minio/minio:RELEASE.2025-07-18T21-56-31Z
     command: server --console-address ":9001"
-    image: docker.io/minio/minio:RELEASE.2025-02-03T21-03-04Z
     healthcheck:
       test: ["CMD", "mc", "ready", "local"]
       interval: 1m30s
@@ -348,17 +379,113 @@ services:
         published: 13245
         host_ip: 127.0.0.1
         protocol: tcp
+        app_protocol: http
       - name: console
         target: 9001
         published: 13246
         host_ip: 127.0.0.1
         protocol: tcp
+        app_protocol: http
+    profiles:
+      - minio
     restart: unless-stopped
+    shm_size: 4mb
     volumes:
       - type: volume
         source: minio_data
         target: /data
+        read_only: false
 
+  zipkin:
+    image: docker.io/openzipkin/zipkin:3
+    healthcheck:
+      test: ["CMD", "wget", "-O", "-", "http://localhost:9411/health"]
+      interval: 1m30s
+      timeout: 5s
+      start_period: 5s
+      start_interval: 2s
+      retries: 5
+    ports:
+      - name: web
+        target: 9411
+        published: 13247
+        host_ip: 127.0.0.1
+        protocol: tcp
+        app_protocol: http
+    profiles:
+      - observability
+    restart: unless-stopped
+    shm_size: 4mb
+
+  loadtest:
+    build:
+      context: ./services/loadtest
+      dockerfile: Dockerfile
+      tags:
+        - adnova-loadtest:latest
+    depends_on:
+      backend:
+        restart: false
+        condition: service_healthy
+        required: true
+    env_file:
+      - path: ./infrastructure/loadtest/.env.template
+        required: true
+      - path: ./infrastructure/loadtest/.env
+        required: false
+    ports:
+      - name: web
+        target: 5001
+        published: 13248
+        host_ip: 127.0.0.1
+        protocol: tcp
+        app_protocol: http
+    profiles:
+      - loadtest
+    restart: unless-stopped
+    shm_size: 4mb
+
+  proxy:
+    image: docker.io/caddy:2-alpine
+    configs:
+      - source: caddy_config
+        target: /etc/caddy/Caddyfile
+    ports:
+      - name: http
+        target: 80
+        published: 80
+        host_ip: 0.0.0.0
+        protocol: tcp
+        app_protocol: http
+      - name: https
+        target: 443
+        published: 443
+        host_ip: 0.0.0.0
+        protocol: tcp
+        app_protocol: http
+      - name: http3
+        target: 443
+        published: 443
+        host_ip: 0.0.0.0
+        protocol: udp
+        app_protocol: http
+    profiles:
+      - proxy
+    restart: unless-stopped
+    shm_size: 4mb
+    volumes:
+      - type: volume
+        source: caddy_data
+        target: /data
+        read_only: false
+      - type: volume
+        source: caddy_config
+        target: /config
+        read_only: false
+      - type: bind
+        source: ./infrastructure/caddy/static
+        target: /var/www
+        read_only: true
 
 volumes:
   redis_data:
@@ -367,21 +494,19 @@ volumes:
   grafana_data:
   prometheus_data:
   minio_data:
+  caddy_data:
+  caddy_config:
 
 configs:
   redis_config:
     file: ./infrastructure/redis/redis.conf
   postgres_config:
     file: ./infrastructure/postgres/postgresql.conf
-  pgadmin_servers:
+  pgadmin_servers_config:
     file: ./infrastructure/pgadmin/servers.json
   grafana_config:
     file: ./infrastructure/grafana/grafana.ini
   prometheus_config:
     file: ./infrastructure/prometheus/prometheus.yaml
-
-secrets:
-  postgres_password:
-    file: ./infrastructure/postgres/password
-  pgadmin_password:
-    file: ./infrastructure/pgadmin/password
+  caddy_config:
+    file: ./infrastructure/caddy/Caddyfile

infrastructure/backend/.env.template

@@ -1,4 +1,4 @@
-DJANGO_SECRET_KEY=secretees
+DJANGO_SECRET_KEY=very_insecure_key
 DJANGO_DEBUG=False
 DJANGO_ALLOWED_HOSTS=*
 DJANGO_CSRF_TRUSTED_ORIGINS=http://localhost,http://127.0.0.1
@@ -21,3 +21,7 @@ MINIO_ENDPOINT=minio:9000
 MINIO_CUSTOM_ENDPOINT_URL=http://127.0.0.1:13244
 MINIO_ACCESS_KEY=admin
 MINIO_SECRET_KEY=password
+
+OTEL_METRICS_EXPORTER=none
+OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://zipkin:9411/api/v2/spans
+OTEL_TRACES_EXPORTER=zipkin_json

infrastructure/caddy/Caddyfile

@@ -0,0 +1,68 @@
+{
+	admin :2019
+
+	metrics {
+		per_host
+	}
+}
+
+(basic-auth) {
+	basic_auth {
+		admin $2a$14$2zQilpLka2h8Sn1mmOLAAezwDN8Zy8Ta36WECk4qt5MTn3CWksR0m
+	}
+}
+
+adnova.itqdev.xyz {
+	@healthPath path /health /health/*
+	handle @healthPath {
+		import basic-auth
+		reverse_proxy http://backend:8080
+	}
+
+	handle_path /static/* {
+		reverse_proxy http://backend-staticfiles:80
+	}
+
+	reverse_proxy http://backend:8080
+}
+
+:8080 {
+	reverse_proxy http://backend:8080
+}
+
+admin.adnova.itqdev.xyz {
+	import basic-auth
+
+	root * /var/www/admin
+	file_server
+}
+
+loadtest.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://loadtest:5001
+}
+
+grafana.adnova.itqdev.xyz {
+	reverse_proxy http://grafana:3000
+}
+
+pgadmin.adnova.itqdev.xyz {
+	reverse_proxy http://pgadmin:80
+}
+
+zipkin.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://zipkin:9411
+}
+
+prometheus.adnova.itqdev.xyz {
+	import basic-auth
+
+	reverse_proxy http://prometheus:9090
+}
+
+minio.adnova.itqdev.xyz {
+	reverse_proxy http://minio:9000
+}

infrastructure/caddy/static/admin/index.html

@@ -0,0 +1,209 @@
+<!DOCTYPE html>
+<html lang="en" class="dark">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>AdNova Admin Resources</title>
+    <script src="https://cdn.tailwindcss.com"></script>
+    <style>
+        body {
+            font-family: 'Inter', sans-serif;
+        }
+
+        @import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');
+    </style>
+</head>
+
+<body class="bg-slate-900 text-gray-300">
+    <div class="container mx-auto p-4 md:p-8">
+        <header class="mb-8">
+            <h1 class="text-4xl font-bold text-white tracking-tight">AdNova Admin Resources Dashboard</h1>
+            <p class="text-indigo-400">Quick access to essential tools and credentials.</p>
+        </header>
+
+        <div id="resources-container" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
+        </div>
+    </div>
+
+    <script>
+        const resources = [
+            {
+                category: "Development Tools",
+                items: [
+                    {
+                        name: "Gitea Repository",
+                        link: "https://git.itqdev.xyz/PROD.2025/AdNova",
+                        description: "Main monorepo on Gitea."
+                    },
+                    {
+                        name: "Rest API Docs",
+                        link: "https://adnova.itqdev.xyz/docs",
+                        description: "Rest API Docs on Swagger."
+                    },
+                    {
+                        name: "Django Admin",
+                        link: "https://adnova.itqdev.xyz/admin/",
+                        username: "admin",
+                        password: "admin",
+                        description: "Django Admin panel."
+                    },
+                    {
+                        name: "Loadtest",
+                        link: "https://loadtest.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Stress test utility with multiple profiles and ability to load mocks."
+                    }
+                ]
+            },
+            {
+                category: "Monitoring",
+                items: [
+                    {
+                        name: "Grafana",
+                        link: "https://grafana.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "proooooood",
+                        description: "Real-time system metrics and dashboards."
+                    },
+                    {
+                        name: "Prometheus",
+                        link: "https://prometheus.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Real-time system metrics and scrape targets."
+                    },
+                    {
+                        name: "Zipkin",
+                        link: "https://zipkin.adnova.itqdev.xyz",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Latest traces from production."
+                    },
+                    {
+                        name: "Django Silk",
+                        link: "https://adnova.itqdev.xyz/silk/",
+                        username: "admin",
+                        password: "admin",
+                        description: "App profiles and performance info in production, need to authenticate first as staff in Django Admin."
+                    },
+                    {
+                        name: "Adnova Alerts",
+                        link: "https://t.me/adnova_alerts",
+                        description: "Telegram channel where alerts are posted."
+                    },
+                    {
+                        name: "Healthcheck",
+                        link: "https://adnova.itqdev.xyz/health",
+                        username: "admin",
+                        password: "kit2025_observability",
+                        description: "Summary of integration status between services."
+                    },
+                    {
+                        name: "Uptime-Kuma Dashboard",
+                        link: "https://status.adnova.itqdev.xyz",
+                        description: "External monitoring of AdNova."
+                    }
+                ]
+            },
+            {
+                category: "Databases",
+                items: [
+                    {
+                        name: "Pgadmin",
+                        link: "https://pgadmin.adnova.itqdev.xyz",
+                        username: "admin@mail.com",
+                        password: "password",
+                        description: "Access to production database."
+                    }
+                ]
+            }
+        ]
+
+        function copyToClipboard ( text )
+        {
+            const textarea = document.createElement( 'textarea' )
+            textarea.value = text
+            document.body.appendChild( textarea )
+            textarea.select()
+            document.execCommand( 'copy' )
+            document.body.removeChild( textarea )
+            alert( 'Copied to clipboard!' )
+        }
+
+        function renderResources ()
+        {
+            const container = document.getElementById( 'resources-container' )
+            container.innerHTML = ''
+
+            resources.forEach( categoryData =>
+            {
+                const categorySection = document.createElement( 'div' )
+                categorySection.className = 'col-span-full mb-4'
+                categorySection.innerHTML = `
+                    <h2 class="text-2xl font-semibold text-white mb-4 border-b border-slate-700 pb-2">
+                        ${ categoryData.category }
+                    </h2>
+                `
+                container.appendChild( categorySection )
+
+                categoryData.items.forEach( resource =>
+                {
+                    const resourceCard = document.createElement( 'div' )
+                    resourceCard.className = 'bg-slate-800 rounded-lg p-6 shadow-lg flex flex-col justify-between'
+
+                    let credentialsHtml = ''
+                    if ( resource.username || resource.password )
+                    {
+                        credentialsHtml = `
+                            <div class="mt-4 pt-4 border-t border-slate-700">
+                                <h4 class="text-sm font-medium text-slate-400 mb-2">Credentials:</h4>
+                                ${ resource.username ? `
+                                    <div class="flex items-center text-sm mb-2">
+                                        <span class="font-semibold text-slate-300 w-24">Username:</span>
+                                        <span id="user-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.username }</span>
+                                        <button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.username }">Copy</button>
+                                    </div>` : '' }
+                                ${ resource.password ? `
+                                    <div class="flex items-center text-sm">
+                                        <span class="font-semibold text-slate-300 w-24">Password:</span>
+                                        <span id="pass-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.password }</span>
+                                        <button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.password }">Copy</button>
+                                    </div>` : '' }
+                            </div>
+                        `
+                    }
+
+                    resourceCard.innerHTML = `
+                        <div>
+                            <h3 class="text-xl font-semibold text-white mb-2">${ resource.name }</h3>
+                            <p class="text-sm text-slate-400 mb-4">${ resource.description || 'No description provided.' }</p>
+                        </div>
+                        <div class="mt-auto">
+                            <a href="${ resource.link }" target="_blank" class="inline-flex items-center justify-center bg-indigo-600 hover:bg-indigo-500 text-white font-bold py-2 px-4 rounded-lg transition-colors duration-300 text-sm">
+                                Go to Resource
+                                <svg class="ml-2 -mr-1 w-4 h-4" fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg"><path d="M11 3a1 1 0 100 2h2.586l-6.293 6.293a1 1 0 101.414 1.414L15 6.414V9a1 1 0 102 0V4a1 1 0 00-1-1h-5z"></path><path d="M5 5a2 2 0 00-2 2v8a2 2 0 002 2h8a2 2 0 002-2v-3a1 1 0 10-2 0v3H5V7h3a1 1 0 000-2H5z"></path></svg>
+                            </a>
+                            ${ credentialsHtml }
+                        </div>
+                    `
+                    container.appendChild( resourceCard )
+                } )
+            } )
+
+            document.querySelectorAll( '.copy-btn' ).forEach( button =>
+            {
+                button.addEventListener( 'click', ( event ) =>
+                {
+                    const textToCopy = event.target.dataset.text
+                    copyToClipboard( textToCopy )
+                } )
+            } )
+        }
+
+        window.onload = renderResources;
+    </script>
+</body>
+
+</html>

infrastructure/grafana/grafana.ini

@@ -48,7 +48,7 @@ domain = localhost
 enforce_domain = false
 
 # The full public facing url
-root_url = %(protocol)s://%(domain)s:%(http_port)s/
+root_url = https://grafana.adnova.itqdev.xyz
 
 # Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
 serve_from_sub_path = false

infrastructure/grafana/provisioning/alerting/contact_points.yaml

@@ -0,0 +1,17 @@
+apiVersion: 1
+
+contactPoints:
+  - orgId: 1
+    name: Telegram
+    receivers:
+      - uid: aet1srtyc40lca
+        type: telegram
+        settings:
+          bottoken: 7797967907:AAGZuUzzuS4LLb525rDNY52Awc2tvpsLjd4
+          chatid: "-1002555823797"
+          disable_notification: false
+          disable_web_page_preview: false
+          message: '{{ template "telegram.default.message" . }}'
+          parse_mode: Markdown
+          protect_content: false
+        disableResolveMessage: false

infrastructure/grafana/provisioning/alerting/rules.yaml

@@ -0,0 +1,141 @@
+apiVersion: 1
+
+groups:
+  - orgId: 1
+    name: Default
+    folder: Backend
+    interval: 10s
+    rules:
+      - uid: aet1xbx1yaupsb
+        title: Backend p99 > 500 ms
+        condition: C
+        data:
+          - refId: A
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: prometheus
+            model:
+              editorMode: code
+              expr: |
+                histogram_quantile(
+                  0.99,
+                  sum(
+                    rate(
+                      caddy_http_request_duration_seconds_bucket{instance="proxy:2019",handler="reverse_proxy",host="proxy:8080",job="caddy"}[$__rate_interval]
+                    )
+                  ) by (le)
+                )
+              instant: true
+              intervalMs: 1000
+              legendFormat: __auto
+              maxDataPoints: 43200
+              range: false
+              refId: A
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              conditions:
+                - evaluator:
+                    params:
+                      - 0.5
+                    type: gte
+                  operator:
+                    type: and
+                  query:
+                    params:
+                      - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+              datasource:
+                type: __expr__
+                uid: __expr__
+              expression: A
+              intervalMs: 1000
+              maxDataPoints: 43200
+              refId: C
+              type: threshold
+        dashboardUid: e3a78c36-2f34-4ad6-81d5-284002896829
+        panelId: 32
+        noDataState: OK
+        execErrState: Error
+        for: 10s
+        keepFiringFor: 10s
+        annotations:
+          __dashboardUid__: e3a78c36-2f34-4ad6-81d5-284002896829
+          __panelId__: "32"
+          runbook_url: https://admin.adnova.itqdev.xyz
+          summary: p99 > 500 ms
+        isPaused: false
+        notification_settings:
+          receiver: Telegram
+  - orgId: 1
+    name: Default
+    folder: Postgres
+    interval: 10s
+    rules:
+      - uid: fet1txr4slywwe
+        title: "> 100 QPS on Postgresql"
+        condition: C
+        data:
+          - refId: A
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: prometheus
+            model:
+              editorMode: code
+              expr: |
+                sum(
+                  irate(pg_stat_database_xact_commit{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
+                )
+                + sum(
+                  irate(pg_stat_database_xact_rollback{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
+                )
+              instant: true
+              intervalMs: 1000
+              legendFormat: __auto
+              maxDataPoints: 43200
+              range: false
+              refId: A
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              conditions:
+                - evaluator:
+                    params:
+                      - 100
+                    type: gte
+                  operator:
+                    type: and
+                  query:
+                    params:
+                      - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+              datasource:
+                type: __expr__
+                uid: __expr__
+              expression: A
+              intervalMs: 1000
+              maxDataPoints: 43200
+              refId: C
+              type: threshold
+        dashboardUid: postgres-overview
+        panelId: 14
+        noDataState: NoData
+        execErrState: Error
+        for: 10s
+        keepFiringFor: 1m
+        annotations:
+          __dashboardUid__: postgres-overview
+          __panelId__: "14"
+          runbook_url: https://admin.adnova.itqdev.xyz
+          summary: Postgresql QPS exceeded 100
+        isPaused: false
+        notification_settings:
+          receiver: Telegram

infrastructure/grafana/provisioning/alerting/templates.yaml

@@ -0,0 +1,54 @@
+apiVersion: 1
+
+templates:
+  - orgId: 1
+    name: Telegram
+    template: |
+      {{ define "telegram.default.message" }} {{ if gt (len .Alerts.Firing) 0 }}
+      🔥🚨 *FIRE IN THE HOLE!* 🚨🔥
+      We've got *{{ len .Alerts.Firing }} firing alert(s)* that need your immediate attention!
+      {{ range .Alerts.Firing }}
+      ---
+      *Alert:* `{{ .Labels.alertname }}`
+      {{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
+      *Status:* 🔴 *FIRING* since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }}
+
+      {{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
+      {{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
+      *Labels:*
+      {{ range .Labels.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}
+      {{ if gt (len .Annotations) 0 }}*Annotations:*
+      {{ range .Annotations.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}{{ end }}
+
+      {{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
+      {{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
+      {{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }})
+      {{ end }}{{ if .SilenceURL }}🤫 [Silence Alert]({{ .SilenceURL }})
+      {{ end }}--- {{ end }} {{ end }}
+      {{ if gt (len .Alerts.Resolved) 0 }}
+      ✅🟢 *ALL CLEAR!* 🟢✅
+      Great news! *{{ len .Alerts.Resolved }} alert(s)* have been resolved.
+      {{ range .Alerts.Resolved }}
+      ---
+      *Alert:* `{{ .Labels.alertname }}`
+      {{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
+      *Status:* ✅ *RESOLVED* at {{ .EndsAt.Format "2006-01-02 15:04:05 MST" }} (was active since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }})
+
+      {{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
+      {{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
+      *Labels:*
+      {{ range .Labels.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}
+      {{ if gt (len .Annotations) 0 }}*Annotations:*
+      {{ range .Annotations.SortedPairs }}  • `{{ .Name }}` = `{{ .Value }}`
+      {{ end }}{{ end }}
+
+      {{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
+      {{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
+      {{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }}) {{ end }} {{ end }} {{ end }}
+      {{ if or (gt (len .Alerts.Firing) 0) (gt (len .Alerts.Resolved) 0) }}
+      🔔 *Grafana Alertmanager:* [View All Alerts]({{ template "__alertmanagerURL" . }}) 🔔
+      {{ end }}
+      {{ end }}

infrastructure/grafana/provisioning/dashboards/Backend/requests-by-view.json

@@ -0,0 +1,910 @@
+{
+    "annotations": {
+        "list": [
+            {
+                "builtIn": 1,
+                "datasource": {
+                    "type": "grafana",
+                    "uid": "-- Grafana --"
+                },
+                "enable": true,
+                "hide": true,
+                "iconColor": "rgba(0, 211, 255, 1)",
+                "name": "Annotations & Alerts",
+                "type": "dashboard"
+            }
+        ]
+    },
+    "description": "A dashboard that monitors Django which focuses on breaking down requests by view.",
+    "editable": true,
+    "fiscalYearStartMonth": 0,
+    "graphTooltip": 0,
+    "id": 12,
+    "links": [
+        {
+            "tags": ["backend"],
+            "targetBlank": true,
+            "title": "Backend Dashboards",
+            "type": "dashboards"
+        }
+    ],
+    "liveNow": true,
+    "panels": [
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 0
+            },
+            "id": 1,
+            "panels": [],
+            "title": "Summary",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "red"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 0.95
+                            },
+                            {
+                                "color": "green",
+                                "value": 0.99
+                            }
+                        ]
+                    },
+                    "unit": "percentunit"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 0,
+                "y": 1
+            },
+            "id": 2,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum(\n  rate(\n    django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\",\n        status!~\"[4-5].*\"\n      }[1w]\n    )\n) /\nsum(\n  rate(\n    django_http_responses_total_by_status_view_method_total{\n      namespace=~\"$namespace\",\n      job=~\"$job\",\n      view=\"$view\",\n      method=~\"$method\"\n    }[1w]\n  )\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Success Rate (non 4xx-5xx responses) [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 10
+                            },
+                            {
+                                "color": "red",
+                                "value": 100
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 6,
+                "y": 1
+            },
+            "id": 3,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum by (view) (\n  increase(\n    django_http_exceptions_total_by_view_total{\n      namespace=~\"$namespace\",\n      job=~\"$job\",\n      view=\"$view\",\n    }[1w]\n  ) > 0\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "HTTP Exceptions [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 1000
+                            },
+                            {
+                                "color": "red",
+                                "value": 2000
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 12,
+                "y": 1
+            },
+            "id": 4,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum (\n    rate (\n        django_http_requests_latency_seconds_by_view_method_bucket {\n          namespace=~\"$namespace\",\n          job=~\"$job\",\n          view=\"$view\",\n          method=~\"$method\"\n        }[$__range]\n    )\n  ) by (job, le)\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Average Request Latency (P50) [1w]",
+            "type": "stat"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "yellow",
+                                "value": 2500
+                            },
+                            {
+                                "color": "red",
+                                "value": 5000
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 4,
+                "w": 6,
+                "x": 18,
+                "y": 1
+            },
+            "id": 5,
+            "options": {
+                "colorMode": "value",
+                "graphMode": "area",
+                "justifyMode": "auto",
+                "orientation": "auto",
+                "percentChangeColorMode": "standard",
+                "reduceOptions": {
+                    "calcs": ["lastNotNull"],
+                    "fields": "",
+                    "values": false
+                },
+                "showPercentChange": false,
+                "textMode": "auto",
+                "wideLayout": true
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum (\n    rate (\n        django_http_requests_latency_seconds_by_view_method_bucket {\n          namespace=~\"$namespace\",\n          job=~\"$job\",\n          view=\"$view\",\n          method=~\"$method\"\n        }[$__range]\n    )\n  ) by (job, le)\n)\n",
+                    "refId": "A"
+                }
+            ],
+            "title": "Average Request Latency (P95) [1w]",
+            "type": "stat"
+        },
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 5
+            },
+            "id": 6,
+            "panels": [],
+            "title": "Request & Responses",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 0,
+                "y": 6
+            },
+            "id": 7,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_requests_total_by_view_transport_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "reqps",
+                    "refId": "A"
+                }
+            ],
+            "title": "Requests",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "percent"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "2xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "green",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "3xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "blue",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "4xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "yellow",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "5xx"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "red",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 12,
+                "y": 6
+            },
+            "id": 8,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"2.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "2xx",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"3.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "3xx",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"4.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "4xx",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        status=~\"5.*\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (job), 0.001\n)\n",
+                    "legendFormat": "5xx",
+                    "refId": "D"
+                }
+            ],
+            "title": "Responses",
+            "type": "timeseries"
+        },
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 14
+            },
+            "id": 9,
+            "panels": [],
+            "title": "Latency & Status Codes",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 100,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "value"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "reqps"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 0,
+                "y": 15
+            },
+            "id": 10,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum(\n    rate(\n      django_http_responses_total_by_status_view_method_total{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\",\n      }[$__rate_interval]\n    ) > 0\n  ) by (namespace, job, view, status, method), 0.001\n)\n",
+                    "legendFormat": "{{ view }} / {{ status }} / {{ method }}",
+                    "refId": "A"
+                }
+            ],
+            "title": "Responses Status Codes",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 10,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "s"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 12,
+                "x": 12,
+                "y": 15
+            },
+            "id": 11,
+            "options": {
+                "legend": {
+                    "calcs": ["lastNotNull", "mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "50 - {{ view }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "95 - {{ view }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.99,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "99 - {{ view }}",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.999,\n  sum(\n    irate(\n      django_http_requests_latency_seconds_by_view_method_bucket{\n        namespace=~\"$namespace\",\n        job=~\"$job\",\n        view=\"$view\",\n        method=~\"$method\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (view, le)\n)\n",
+                    "legendFormat": "99.9 - {{ view }}",
+                    "refId": "D"
+                }
+            ],
+            "title": "Request Latency",
+            "type": "timeseries"
+        }
+    ],
+    "preload": false,
+    "refresh": "5s",
+    "schemaVersion": 41,
+    "tags": ["backend"],
+    "templating": {
+        "list": [
+            {
+                "label": "Data source",
+                "name": "datasource",
+                "query": "prometheus",
+                "refresh": 1,
+                "type": "datasource"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Namespace",
+                "name": "namespace",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{}, namespace)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Job",
+                "name": "job",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\"}, job)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "View",
+                "name": "view",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view!~\"<unnamed view>|health_check:health_check_home|prometheus-django-metrics\"}, view)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": true,
+                "label": "Method",
+                "multi": true,
+                "name": "method",
+                "query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view=~\"$view\"}, method)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            }
+        ]
+    },
+    "time": {
+        "from": "now-30m",
+        "to": "now"
+    },
+    "timepicker": {},
+    "timezone": "browser",
+    "title": "Requests / By View",
+    "uid": "backend-requests-by-view",
+    "version": 2,
+    "weekStart": "monday"
+}

infrastructure/grafana/provisioning/dashboards/Celery/by-task.json

@@ -0,0 +1,768 @@
+{
+    "annotations": {
+        "list": [
+            {
+                "builtIn": 1,
+                "datasource": {
+                    "type": "grafana",
+                    "uid": "-- Grafana --"
+                },
+                "enable": true,
+                "hide": true,
+                "iconColor": "rgba(0, 211, 255, 1)",
+                "name": "Annotations & Alerts",
+                "type": "dashboard"
+            }
+        ]
+    },
+    "description": "A dashboard that monitors Celery.",
+    "editable": true,
+    "fiscalYearStartMonth": 0,
+    "graphTooltip": 0,
+    "id": 2,
+    "links": [
+        {
+            "tags": ["celery"],
+            "targetBlank": true,
+            "title": "Celery Dashboards",
+            "type": "dashboards"
+        }
+    ],
+    "liveNow": true,
+    "panels": [
+        {
+            "collapsed": false,
+            "gridPos": {
+                "h": 1,
+                "w": 24,
+                "x": 0,
+                "y": 0
+            },
+            "id": 1,
+            "panels": [],
+            "title": "Tasks",
+            "type": "row"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "align": "auto",
+                        "cellOptions": {
+                            "type": "auto"
+                        },
+                        "inspect": false
+                    },
+                    "mappings": [],
+                    "noValue": 0,
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "Success Rate"
+                        },
+                        "properties": [
+                            {
+                                "id": "unit",
+                                "value": "percentunit"
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 16,
+                "x": 0,
+                "y": 1
+            },
+            "id": 2,
+            "options": {
+                "cellHeight": "sm",
+                "footer": {
+                    "countRows": false,
+                    "enablePagination": true,
+                    "fields": "",
+                    "reducer": ["sum"],
+                    "show": false
+                },
+                "showHeader": true,
+                "sortBy": [
+                    {
+                        "desc": true,
+                        "displayName": "Succeeded"
+                    }
+                ]
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n/(sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n+sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name)\n) > -1\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_sent_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "D"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_received_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "E"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_rejected_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "F"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_retried_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "G"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_revoked_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  )\n) by (name) > 0\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "H"
+                }
+            ],
+            "title": "Task Stats",
+            "transformations": [
+                {
+                    "id": "merge"
+                },
+                {
+                    "id": "organize",
+                    "options": {
+                        "excludeByName": {
+                            "Time": true
+                        },
+                        "indexByName": {
+                            "Value #A": 1,
+                            "Value #B": 2,
+                            "Value #C": 3,
+                            "Value #D": 4,
+                            "Value #E": 5,
+                            "Value #F": 6,
+                            "Value #G": 7,
+                            "Value #H": 8,
+                            "name": 0
+                        },
+                        "renameByName": {
+                            "Value #A": "Success Rate",
+                            "Value #B": "Succeeded",
+                            "Value #C": "Failed",
+                            "Value #D": "Sent",
+                            "Value #E": "Received",
+                            "Value #F": "Rejected",
+                            "Value #G": "Retried",
+                            "Value #H": "Revoked",
+                            "name": "Name"
+                        }
+                    }
+                }
+            ],
+            "type": "table"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "align": "auto",
+                        "cellOptions": {
+                            "type": "auto"
+                        },
+                        "inspect": false
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 8,
+                "x": 16,
+                "y": 1
+            },
+            "id": 3,
+            "options": {
+                "cellHeight": "sm",
+                "footer": {
+                    "countRows": false,
+                    "enablePagination": true,
+                    "fields": "",
+                    "reducer": ["sum"],
+                    "show": false
+                },
+                "showHeader": true,
+                "sortBy": [
+                    {
+                        "desc": true,
+                        "displayName": "Value"
+                    }
+                ]
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "round(\n  sum (\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__range]\n    )\n  ) by (name, exception) > 0\n)\n",
+                    "format": "table",
+                    "instant": true,
+                    "refId": "A"
+                }
+            ],
+            "title": "Task Exceptions",
+            "transformations": [
+                {
+                    "id": "organize",
+                    "options": {
+                        "excludeByName": {
+                            "Time": true,
+                            "job": true
+                        },
+                        "indexByName": {
+                            "Value": 2,
+                            "exception": 1,
+                            "name": 0
+                        },
+                        "renameByName": {
+                            "exception": "Exception",
+                            "name": "Task"
+                        }
+                    }
+                }
+            ],
+            "type": "table"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 0,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 9
+            },
+            "id": 4,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_succeeded_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Succeeded - {{ name }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Failed - {{ name }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_sent_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Sent - {{ name }}",
+                    "refId": "C"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_received_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Received - {{ name }}",
+                    "refId": "D"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_retried_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Retried - {{ name }}",
+                    "refId": "E"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_revoked_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Revoked - {{ name }}",
+                    "refId": "F"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_rejected_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name) > 0\n",
+                    "legendFormat": "Rejected - {{ name }}",
+                    "refId": "G"
+                }
+            ],
+            "title": "Tasks Completed",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "color": {
+                        "mode": "palette-classic"
+                    },
+                    "custom": {
+                        "axisBorderShow": false,
+                        "axisCenteredZero": false,
+                        "axisColorMode": "text",
+                        "axisLabel": "",
+                        "axisPlacement": "auto",
+                        "barAlignment": 0,
+                        "barWidthFactor": 0.6,
+                        "drawStyle": "line",
+                        "fillOpacity": 0,
+                        "gradientMode": "none",
+                        "hideFrom": {
+                            "legend": false,
+                            "tooltip": false,
+                            "viz": false
+                        },
+                        "insertNulls": false,
+                        "lineInterpolation": "linear",
+                        "lineWidth": 1,
+                        "pointSize": 5,
+                        "scaleDistribution": {
+                            "type": "linear"
+                        },
+                        "showPoints": "auto",
+                        "spanNulls": false,
+                        "stacking": {
+                            "group": "A",
+                            "mode": "none"
+                        },
+                        "thresholdsStyle": {
+                            "mode": "off"
+                        }
+                    },
+                    "mappings": [],
+                    "thresholds": {
+                        "mode": "absolute",
+                        "steps": [
+                            {
+                                "color": "green"
+                            },
+                            {
+                                "color": "red",
+                                "value": 80
+                            }
+                        ]
+                    },
+                    "unit": "short"
+                },
+                "overrides": []
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 17
+            },
+            "id": 5,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "hideZeros": false,
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "12.0.2",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "sum (\n  round(\n    increase(\n      celery_task_failed_total{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    )\n  )\n) by (name, exception) > 0\n",
+                    "legendFormat": "{{ name }}/{{ exception }}",
+                    "refId": "A"
+                }
+            ],
+            "title": "Task Exceptions",
+            "type": "timeseries"
+        },
+        {
+            "datasource": {
+                "type": "datasource",
+                "uid": "-- Mixed --"
+            },
+            "fieldConfig": {
+                "defaults": {
+                    "custom": {
+                        "spanNulls": false
+                    },
+                    "unit": "s"
+                },
+                "overrides": [
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P50"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "green",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P95"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "yellow",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "matcher": {
+                            "id": "byName",
+                            "options": "P99"
+                        },
+                        "properties": [
+                            {
+                                "id": "color",
+                                "value": {
+                                    "fixedColor": "red",
+                                    "mode": "fixed"
+                                }
+                            }
+                        ]
+                    }
+                ]
+            },
+            "gridPos": {
+                "h": 8,
+                "w": 24,
+                "x": 0,
+                "y": 25
+            },
+            "id": 6,
+            "options": {
+                "legend": {
+                    "calcs": ["mean", "max"],
+                    "displayMode": "table",
+                    "placement": "right",
+                    "showLegend": true,
+                    "sortBy": "Mean",
+                    "sortDesc": true
+                },
+                "tooltip": {
+                    "mode": "multi",
+                    "sort": "desc"
+                }
+            },
+            "pluginVersion": "v11.1.0",
+            "targets": [
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.50,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P50 - {{ name }}",
+                    "refId": "A"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.95,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P95 - {{ name }}",
+                    "refId": "B"
+                },
+                {
+                    "datasource": {
+                        "type": "prometheus",
+                        "uid": "$datasource"
+                    },
+                    "expr": "histogram_quantile(0.99,\n  sum(\n    irate(\n      celery_task_runtime_bucket{\n        job=\"$job\",\n        name=~\"$task\",\n        queue_name=~\"$queue_name\"\n      }[$__rate_interval]\n    ) > 0\n  ) by (name, job, le)\n)\n",
+                    "legendFormat": "P99 - {{ name }}",
+                    "refId": "C"
+                }
+            ],
+            "title": "Tasks Runtime",
+            "type": "timeseries"
+        }
+    ],
+    "preload": false,
+    "refresh": "10s",
+    "schemaVersion": 41,
+    "tags": ["celery"],
+    "templating": {
+        "list": [
+            {
+                "label": "Data source",
+                "name": "datasource",
+                "query": "prometheus",
+                "refresh": 1,
+                "type": "datasource"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Namespace",
+                "name": "namespace",
+                "query": "label_values(celery_worker_up{}, namespace)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Job",
+                "name": "job",
+                "query": "label_values(celery_worker_up{namespace=\"$namespace\"}, job)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Queue Name",
+                "name": "queue_name",
+                "query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", name!~\"None\"}, queue_name)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            },
+            {
+                "datasource": {
+                    "type": "prometheus",
+                    "uid": "${datasource}"
+                },
+                "includeAll": false,
+                "label": "Task",
+                "multi": true,
+                "name": "task",
+                "query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", queue_name=~\"$queue_name\", name!~\"None\"}, name)",
+                "refresh": 2,
+                "sort": 1,
+                "type": "query"
+            }
+        ]
+    },
+    "time": {
+        "from": "now-24h",
+        "to": "now"
+    },
+    "timepicker": {},
+    "timezone": "browser",
+    "title": "By Task",
+    "uid": "celery-tasks-by-task",
+    "version": 1,
+    "weekStart": "monday"
+}

infrastructure/grafana/provisioning/dashboards/providers.yaml

@@ -1,11 +1,13 @@
 apiVersion: 1
 
 providers:
-  - name: "celery"
+  - name: "default"
     orgId: 1
     folder: ""
     type: file
+    disableDeletion: false
     updateIntervalSeconds: 10
+    allowUiUpdates: false
     options:
       path: /etc/grafana/provisioning/dashboards
       foldersFromFilesStructure: true

infrastructure/grafana/provisioning/datasources/datasources.yaml

@@ -1,10 +1,29 @@
 apiVersion: 1
 
 datasources:
-  - name: Infinity
+  - name: Backend | Infinity
     type: yesoreyeram-infinity-datasource
     access: proxy
     orgId: 1
     uid: infinity
+    url: http://backend:8080
+    basicAuth: false
+    basicAuthUser: ""
+    withCredentials: false
     isDefault: false
     editable: false
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    orgId: 1
+    uid: prometheus
+    url: http://prometheus:9090
+    basicAuth: false
+    basicAuthUser: ""
+    withCredentials: false
+    isDefault: true
+    editable: false
+    jsonData:
+      httpMethod: POST
+      queryTimeout: 10s
+      timeInterval": 10s

infrastructure/loadtest/.env.template

@@ -0,0 +1 @@
+BACKEND_ADDRESS=http://backend:8080

infrastructure/pgadmin/.env.template

@@ -1,2 +1,4 @@
 PGADMIN_DEFAULT_EMAIL=admin@mail.com
-PGADMIN_DEFAULT_PASSWORD_FILE=/run/secrets/pgadmin_password
+PGADMIN_DEFAULT_PASSWORD=password
+PGADMIN_DISABLE_POSTFIX=True
+PGADMIN_REPLACE_SERVERS_ON_STARTUP=True

infrastructure/postgres/.env.template

@@ -1,3 +1,3 @@
 POSTGRES_DB=postgres
 POSTGRES_USER=postgres
-POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
+POSTGRES_PASSWORD=postgres

infrastructure/prometheus/prometheus.yaml

@@ -1,7 +1,7 @@
 # Global config
 global:
-  scrape_interval: 10s
-  scrape_timeout: 10s
+  scrape_interval: 5s
+  scrape_timeout: 5s
   evaluation_interval: 10m
   external_labels:
     environment: local
@@ -10,24 +10,31 @@ global:
 scrape_configs:
   # Prometheus
   - job_name: prometheus
-    scrape_interval: 5s
     static_configs:
       - targets: ["localhost:9090"]
 
   # Postgres
   - job_name: postgres
-    scrape_interval: 10s
     static_configs:
       - targets: ["postgres-exporter:9187"]
 
   # Redis
   - job_name: redis
-    scrape_interval: 10s
     static_configs:
       - targets: ["redis-exporter:9121"]
 
   # Celery
   - job_name: celery
-    scrape_interval: 30s
+    scrape_interval: 15s
     static_configs:
       - targets: ["celery-exporter:9808"]
+
+  # Backend
+  - job_name: backend
+    static_configs:
+      - targets: ["backend:8080"]
+
+  # Caddy
+  - job_name: caddy
+    static_configs:
+      - targets: ["proxy:2019"]

infrastructure/telegram_bot/.env.template

@@ -1,4 +1,4 @@
 AIOGRAM_BOT_TOKEN=
 AIOGRAM_BACKEND_URL=http://backend:8080
 REDIS_URI=redis://redis:6379
-MINIO_ENDPOINT=minio:9000
+MINIO_ENDPOINT=http://minio:9000

services/backend/.dockerignore

@@ -183,3 +183,6 @@ Dockerfile.staticfiles
 
 # Collected static files
 static
+
+# Profile files
+*.prof

services/backend/.gitignore

@@ -171,3 +171,6 @@ cython_debug/
 
 # Collected static files
 static
+
+# Profile files
+*.prof

services/backend/Dockerfile

@@ -1,7 +1,7 @@
 # Stage 1: Install dependencies
-FROM docker.io/python:3.11-alpine3.20 AS builder
+FROM docker.io/python:3.13-alpine3.22 AS builder
 
-COPY --from=ghcr.io/astral-sh/uv:0.4.30 /uv /uvx /bin/
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
 
 WORKDIR /app
 
@@ -17,7 +17,7 @@ RUN uv sync --no-dev --no-install-project --no-cache
 
 
 # Stage 2: Start the application
-FROM docker.io/python:3.11-alpine3.20
+FROM docker.io/python:3.13-alpine3.22
 
 WORKDIR /app
 
@@ -32,11 +32,12 @@ USER app
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
     PYTHONOPTIMIZE=2 \
-    PATH="/opt/venv/bin:$PATH"
+    PATH="/opt/venv/bin:$PATH" \
+    DJANGO_SETTINGS_MODULE=config.settings
 
 EXPOSE 8080
 
 HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --start-interval=2s --retries=3 \
     CMD wget --no-verbose --tries=1 --spider http://127.0.0.1:8080/health?format=json || exit 1
 
-CMD gunicorn config.wsgi --workers=8 -b 0.0.0.0:8080 --access-logfile - --error-logfile -
+CMD [ "opentelemetry-instrument", "--service_name", "backend-django", "--traces_exporter", "zipkin_json", "gunicorn", "config.wsgi", "--workers=2", "-b", "0.0.0.0:8080", "--access-logfile", "-", "--error-logfile", "-" ]

services/backend/Dockerfile.staticfiles

@@ -1,7 +1,7 @@
 # Stage 1: Install dependencies and compile staticfiles
-FROM docker.io/python:3.11-alpine3.20 AS builder
+FROM docker.io/python:3.13-alpine3.22 AS builder
 
-COPY --from=ghcr.io/astral-sh/uv:0.4.30 /uv /uvx /bin/
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
 
 WORKDIR /app
 
@@ -17,11 +17,12 @@ RUN uv sync --no-dev --no-install-project --no-cache
 
 COPY . .
 
-RUN uv run python manage.py collectstatic --noinput
+RUN uv run --no-dev python manage.py collectstatic --noinput
+
 
 # Stage 2: Start nginx and serve staticfiles
-FROM docker.io/nginx:latest
+FROM docker.io/nginx:1.29-alpine-slim
 
 COPY --from=builder /app/static /usr/share/nginx/html
 
-CMD ["nginx", "-g", "daemon off;"]
+CMD [ "nginx", "-g", "daemon off;" ]

services/backend/README.md

@@ -4,9 +4,9 @@
 
 Ensure you have the following installed on your system:
 
-- [Python](https://www.python.org/) (>=3.10,<3.12)
-- [uv](https://docs.astral.sh/uv/)
-- [Docker](https://www.docker.com/) (for containerized setup)
+- [Python](https://www.python.org/) (>=3.10,<3.14)
+- [uv](https://docs.astral.sh/uv/) (latest version recommended)
+- [Docker](https://www.docker.com/) (for containerized setup, latest version recommended)
 
 ## Basic setup
 

services/backend/api/v1/ads/views.py

@@ -4,6 +4,7 @@ from uuid import UUID
 from django.http import Http404, HttpRequest
 from django.shortcuts import get_object_or_404
 from ninja import Router
+from silk.profiling.profiler import silk_profile
 
 from api.v1 import schemas as global_schemas
 from api.v1.ads import schemas
@@ -21,6 +22,7 @@ router = Router(tags=["ads"])
         status.NOT_FOUND: global_schemas.NotFoundError,
     },
 )
+@silk_profile("Get Advertisment")
 def get_advertisment(
     request: HttpRequest, client_id: UUID
 ) -> tuple[status, Campaign]:

services/backend/api/v1/generate/views.py

@@ -49,8 +49,12 @@ def get_generate_ad_text_result(
     if task_result.status == celery.states.PENDING:
         raise Http404
 
+    result = task_result.result
+    if task_result.status != celery.states.SUCCESS:
+        result = None
+
     return status.OK, schemas.Promise(
         task_id=task_result.task_id,
         status=task_result.status,
-        result=task_result.result,
+        result=result,
     )

services/backend/api/v1/time/__init__.py

@@ -1 +0,0 @@
-# noqa: A005

services/backend/apps/advertiser/models.py

@@ -41,10 +41,10 @@ class Advertiser(BaseModel):
             (
                 Decimal(str(total_clicks))
                 / Decimal(str(total_impressions))
-                * Decimal("100")
+                * Decimal(100)
             )
             if total_impressions > 0
-            else Decimal("0")
+            else Decimal(0)
         )
 
         return {
@@ -115,9 +115,9 @@ class Advertiser(BaseModel):
             conversion = (
                 Decimal(str(metrics["clicks_count"]))
                 / Decimal(str(metrics["impressions_count"]))
-                * Decimal("100")
+                * Decimal(100)
                 if metrics["impressions_count"] > 0
-                else Decimal("0")
+                else Decimal(0)
             )
 
             daily_stats.append(

services/backend/apps/campaign/models.py

@@ -256,10 +256,10 @@ class Campaign(BaseModel):
             (
                 Decimal(str(clicks_count))
                 / Decimal(str(impressions_count))
-                * Decimal("100")
+                * Decimal(100)
             )
             if impressions_count > 0
-            else Decimal("0")
+            else Decimal(0)
         )
         spent_impressions = Decimal(str(impressions.get("spent", 0) or 0))
         spent_clicks = Decimal(str(clicks.get("spent", 0) or 0))

services/backend/config/settings.py

@@ -4,6 +4,7 @@ import contextlib
 import logging
 from collections.abc import Callable
 from pathlib import Path
+from typing import TYPE_CHECKING
 
 import django_stubs_ext
 import environ
@@ -12,6 +13,10 @@ from health_check.plugins import plugin_dir
 
 from integrations.yandexai.healthcheck import YandexAIHealthCheck
 
+if TYPE_CHECKING:
+    from django.contrib.auth.models import User
+
+
 BASE_DIR = Path(__file__).resolve().parent.parent
 
 env = environ.Env()
@@ -53,7 +58,7 @@ REDIS_URI = env("REDIS_URI", default="redis://localhost:6379")
 
 CACHES = {
     "default": {
-        "BACKEND": "django.core.cache.backends.redis.RedisCache",
+        "BACKEND": "django_prometheus.cache.backends.redis.RedisCache",
         "LOCATION": REDIS_URI,
         "TIMEOUT": None,
         "KEY_PREFIX": "backend",
@@ -80,6 +85,9 @@ CELERY_TASK_TRACK_STARTED = True
 # Database
 
 DB_URI = env.db_url("DJANGO_DB_URI", default="sqlite:///db.sqlite3")
+DB_URI["ENGINE"] = DB_URI["ENGINE"].replace(
+    "django.db.backends", "django_prometheus.db.backends"
+)
 
 DATABASES = {"default": {**DB_URI, "CONN_MAX_AGE": 50}}
 
@@ -284,12 +292,15 @@ INTERNAL_IPS = env(
 )
 
 MIDDLEWARE = [
+    "silk.middleware.SilkyMiddleware",
+    "django_prometheus.middleware.PrometheusBeforeMiddleware",
     "django_guid.middleware.guid_middleware",
     "corsheaders.middleware.CorsMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
     "django.middleware.csrf.CsrfViewMiddleware",
     "django.contrib.auth.middleware.AuthenticationMiddleware",
     "django.contrib.messages.middleware.MessageMiddleware",
+    "django_prometheus.middleware.PrometheusAfterMiddleware",
 ]
 
 SIGNING_BACKEND = "django.core.signing.TimestampSigner"
@@ -442,8 +453,10 @@ INSTALLED_APPS = [
     "corsheaders",
     "django_extensions",
     "django_guid",
+    "django_prometheus",
     "ninja",
     "minio_storage",
+    "silk",
     # Internal apps
     "apps.core",
     "apps.advertiser",
@@ -516,6 +529,13 @@ SECRET_KEY = env("DJANGO_SECRET_KEY", default="very_insecure_key")
 SECRET_KEY_FALLBACKS: list[str] = []
 
 
+# Auth
+
+LOGIN_REDIRECT_URL = "/admin/"
+
+LOGIN_URL = "/admin/"
+
+
 # Sessions
 
 SESSION_CACHE_ALIAS = "default"
@@ -594,3 +614,63 @@ DEBUG_TOOLBAR_CONFIG = {"SHOW_COLLAPSED": True, "UPDATE_ON_FETCH": True}
 if DEBUG and DEBUG_TOOLBAR_ENABLED:
     INSTALLED_APPS.append("debug_toolbar")
     MIDDLEWARE.append("debug_toolbar.middleware.DebugToolbarMiddleware")
+
+
+# Prometheus
+
+PROMETHEUS_LATENCY_BUCKETS = (
+    0.005,
+    0.01,
+    0.025,
+    0.05,
+    0.075,
+    0.1,
+    0.25,
+    0.5,
+    0.75,
+    1.0,
+    2.5,
+    5.0,
+    7.5,
+    10.0,
+    25.0,
+    50.0,
+    75.0,
+    float("inf"),
+)
+
+
+# django-silk
+
+SILKY_PYTHON_PROFILER = True
+
+SILKY_PYTHON_PROFILER_BINARY = True
+
+SILKY_PYTHON_PROFILER_RESULT_PATH = "./profiles"
+
+SILKY_PYTHON_PROFILER_EXTENDED_FILE_NAME = True
+
+SILKY_AUTHENTICATION = True
+
+SILKY_AUTHORISATION = True
+
+
+def is_allowed_to_use_profiling(user: "User") -> bool:
+    return user.is_staff
+
+
+SILKY_PERMISSIONS = is_allowed_to_use_profiling
+
+SILKY_MAX_RECORDED_REQUESTS = 10**3
+
+SILKY_MAX_RECORDED_REQUESTS_CHECK_PERCENT = 10
+
+SILKY_MAX_REQUEST_BODY_SIZE = 128
+
+SILKY_INTERCEPT_PERCENT = 25
+
+SILKY_META = True
+
+SILKY_DYNAMIC_PROFILING = [
+    {"module": "api.v1.ads.views", "function": "get_advertisment"}
+]

services/backend/config/urls.py

@@ -16,6 +16,10 @@ urlpatterns = [
     path("admin/", admin.site.urls),
     # API urls
     path("", include("api.urls")),
+    # Prometheus urls
+    path("", include("django_prometheus.urls")),
+    # Django-silk
+    path("silk/", include("silk.urls", namespace="silk")),
 ]
 
 

services/backend/integrations/yandexai/moderation.py

@@ -1,4 +1,4 @@
-# ruff: noqa: E501, W291
+# ruff: noqa: E501
 import logging
 
 from django.conf import settings

services/backend/manage.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+# ruff: noqa: PLC0415
 """Django's command-line utility for administrative tasks."""
 
 import os

services/backend/pyproject.toml

@@ -1,38 +1,58 @@
 [project]
-name = "adnova-backend"
-version = "0.1.0"
-readme = "README.md"
-requires-python = ">=3.10,<3.12"
 dependencies = [
- "celery>=5.4.0",
- "colorlog>=6.9.0",
- "django-cors-headers>=4.6.0",
- "django-environ>=0.11.2",
- "django-extensions>=3.2.3",
- "django-guid>=3.5.0",
- "django-health-check>=3.18.3",
- "django-minio-storage>=0.5.7",
- "django-ninja>=1.3.0",
- "django-stubs-ext>=5.1.3",
- "gunicorn>=23.0.0",
- "httpx>=0.28.1",
- "pillow>=11.1.0",
- "psycopg2-binary>=2.9.10",
- "pydantic>=2.10.5",
- "pyjwt>=2.10.1",
- "python-json-logger>=3.2.1",
- "pytz>=2024.2",
- "redis>=5.2.1",
- "yandex-cloud-ml-sdk>=0.3.1",
+ "celery>=5.5.0,<6.0.0",
+ "colorlog>=6.9.0,<7.0.0",
+ "django-cors-headers>=4.7.0,<5.0.0",
+ "django-environ>=0.12.0,<1.0.0",
+ "django-extensions>=4.1.0,<5.0.0",
+ "django-guid>=3.5.1,<4.0.0",
+ "django-health-check>=3.18.3,<4.0.0",
+ "django-minio-storage>=0.5.7,<0.6.0",
+ "django-ninja>=1.3.0,<2.0.0",
+ "django-prometheus>=2.4.1,<3.0.0",
+ "django-redis>=6.0.0,<7.0.0",
+ "django-silk[formatting]>=5.4.0,<6.0.0",
+ "django-stubs-ext>=5.1.3,<6.0.0",
+ "gunicorn>=23.0.0,<24.0.0",
+ "httpx>=0.28.1,<0.29.0",
+ "opentelemetry-api>=1.35.0",
+ "opentelemetry-distro>=0.56b0",
+ "opentelemetry-exporter-otlp>=1.35.0",
+ "opentelemetry-exporter-zipkin-proto-http>=1.11.1",
+ "opentelemetry-instrumentation-asyncio>=0.56b0",
+ "opentelemetry-instrumentation-celery>=0.56b0",
+ "opentelemetry-instrumentation-dbapi>=0.56b0",
+ "opentelemetry-instrumentation-django>=0.56b0",
+ "opentelemetry-instrumentation-httpx>=0.56b0",
+ "opentelemetry-instrumentation-psycopg2>=0.56b0",
+ "opentelemetry-instrumentation-requests>=0.56b0",
+ "opentelemetry-instrumentation-sqlite3>=0.56b0",
+ "opentelemetry-instrumentation-threading>=0.56b0",
+ "opentelemetry-instrumentation-urllib>=0.56b0",
+ "opentelemetry-instrumentation-urllib3>=0.56b0",
+ "opentelemetry-instrumentation-wsgi>=0.56b0",
+ "opentelemetry-sdk>=1.35.0",
+ "pillow>=11.1.0,<12.0.0",
+ "psycopg2-binary>=2.9.10,<3.0.0",
+ "pydantic>=2.10.5,<3.0.0",
+ "pyjwt>=2.10.1,<3.0.0",
+ "python-json-logger>=3.2.1,<4.0.0",
+ "pytz>=2024.2,<2025.0",
+ "redis>=6.2.0,<7.0.0",
+ "yandex-cloud-ml-sdk>=0.3.1,<0.4.0",
 ]
+name = "adnova-backend"
+readme = "README.md"
+requires-python = ">=3.10,<3.14"
+version = "0.1.0"
 
 [dependency-groups]
 dev = [
-    "coverage>=7.6.12",
-    "django-debug-toolbar>=4.4.6",
-    "django-stubs[compatible-mypy]>=5.1.3",
-    "mypy>=1.15.0",
-    "ruff>=0.9.3",
+ "coverage",
+ "django-debug-toolbar>=5.2,<5.3",
+ "django-stubs[compatible-mypy]",
+ "mypy",
+ "ruff",
 ]
 
 [tool.ruff]
@@ -66,7 +86,7 @@ extend-include = []
 fix = false
 fix-only = false
 force-exclude = true
-include = ["*.py", "*.pyi", "*.ipynb", "**/pyproject.toml"]
+include = ["**/pyproject.toml", "*.ipynb", "*.py", "*.pyi"]
 indent-width = 4
 line-length = 79
 namespace-packages = []
@@ -81,20 +101,20 @@ unsafe-fixes = false
 
 [tool.ruff.analyze]
 detect-string-imports = true
-direction = "Dependencies"
-exclude = []
-include-dependencies = {}
-preview = false
+direction             = "Dependencies"
+exclude               = []
+include-dependencies  = {}
+preview               = false
 
 [tool.ruff.format]
-docstring-code-format = true
+docstring-code-format      = true
 docstring-code-line-length = 79
-exclude = []
-indent-style = "space"
-line-ending = "lf"
-preview = false
-quote-style  = "double"
-skip-magic-trailing-comma = false
+exclude                    = []
+indent-style               = "space"
+line-ending                = "lf"
+preview                    = false
+quote-style                = "double"
+skip-magic-trailing-comma  = false
 
 [tool.ruff.lint]
 allowed-confusables = ["ℹ"]
@@ -109,26 +129,26 @@ extend-unsafe-fixes = []
 external = []
 fixable = ["ALL"]
 ignore = [
-    "ARG",
-    "D",
-    "ANN401",
-    "COM812",
-    "DJ001",
-    "DJ007",
-    "FBT001",
-    "FBT002",
-    "N813",
-    "PLR2004",
-    "PT009",
-    "PT027",
-    "RUF001",
-    "S311",
+ "ANN401",
+ "ARG",
+ "COM812",
+ "D",
+ "DJ001",
+ "DJ007",
+ "FBT001",
+ "FBT002",
+ "N813",
+ "PLR2004",
+ "PT009",
+ "PT027",
+ "RUF001",
+ "S311",
 ]
 logger-objects = []
 per-file-ignores = {}
 preview = false
 select = ["ALL"]
-task-tags = ["TODO", "FIXME", "HACK", "WORKOUT"]
+task-tags = ["FIXME", "HACK", "TODO", "WORKOUT"]
 typing-modules = []
 unfixable = []
 
@@ -136,26 +156,26 @@ unfixable = []
 max-args = 6
 
 [tool.mypy]
-plugins = ["mypy_django_plugin.main"]
 ignore_missing_imports = true
-strict = false
-show_error_context = false
-no_implicit_optional = false
+no_implicit_optional   = false
+plugins                = ["mypy_django_plugin.main"]
+show_error_context     = false
+strict                 = false
 
 [tool.django-stubs]
 django_settings_module = "config.settings"
-strict_settings = false
+strict_settings        = false
 
 [tool.coverage.run]
 omit = [
-    "manage.py",
-    "config/wsgi.py",
-    "config/asgi.py",
-    "config/urls.py",
-    "config/settings.py",
-    "config/handlers.py",
-    "config/errors.py",
-    "integrations/yandexai/*"
+ "config/asgi.py",
+ "config/errors.py",
+ "config/handlers.py",
+ "config/settings.py",
+ "config/urls.py",
+ "config/wsgi.py",
+ "integrations/yandexai/*",
+ "manage.py",
 ]
 
 [tool.coverage.report]

services/backend/scripts/initdb

@@ -11,3 +11,4 @@ if [ "$DJANGO_CREATE_SUPERUSER" = "True" ]; then
 fi
 
 python manage.py init_cache
+python manage.py silk_clear_request_log

services/loadtest/.dockerignore

@@ -0,0 +1,38 @@
+# If you prefer the allow list template instead of the deny list, see community template:
+# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
+#
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+
+# Test binary, built with `go test -c`
+*.test
+
+# Code coverage profiles and other test artifacts
+*.out
+coverage.*
+*.coverprofile
+profile.cov
+
+# Dependency directories (remove the comment below to include it)
+# vendor/
+
+# Go workspace file
+go.work
+go.work.sum
+
+# env file
+.env
+
+# Editor/IDE
+# .idea/
+# .vscode/
+
+# App binary
+main
+
+# Gitignore
+.gitignore

services/loadtest/.env.template

@@ -0,0 +1,4 @@
+# Change all vars before going to production and remove all comments (!)
+# Below all environment variables and default values
+
+BACKEND_ADDRESS=http://localhost:8080

services/loadtest/.gitignore

@@ -0,0 +1,35 @@
+# If you prefer the allow list template instead of the deny list, see community template:
+# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
+#
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+
+# Test binary, built with `go test -c`
+*.test
+
+# Code coverage profiles and other test artifacts
+*.out
+coverage.*
+*.coverprofile
+profile.cov
+
+# Dependency directories (remove the comment below to include it)
+# vendor/
+
+# Go workspace file
+go.work
+go.work.sum
+
+# env file
+.env
+
+# Editor/IDE
+# .idea/
+# .vscode/
+
+# App binary
+main

services/loadtest/Dockerfile

@@ -0,0 +1,28 @@
+ARG GOARCH=amd64
+
+# Stage 1: Build go binary
+FROM docker.io/golang:1.24-alpine AS build
+
+WORKDIR /build
+
+COPY . .
+
+RUN go mod download
+
+ARG GOARCH
+
+RUN CGO_ENABLED=0 GOARCH=$GOARCH go build -o loadtest
+
+
+# Stage 2: Run go binary
+FROM docker.io/alpine:3.22
+
+WORKDIR /app
+
+COPY --from=build /build/loadtest .
+
+COPY . .
+
+EXPOSE 5001
+
+CMD [ "./loadtest", "--port", "5001" ]

services/loadtest/README.md

@@ -0,0 +1,68 @@
+# AdNova Loadtest
+
+## Prerequisites
+
+Ensure you have the following installed on your system:
+
+- [Go](https://go.dev/) (1.24 recommended)
+- [Docker](https://www.docker.com/) (for containerized setup, latest version recommended)
+
+## Basic setup
+
+### Installation
+
+#### Clone the project
+
+#### Go to the project directory
+
+```bash
+cd AdNova/services/loadtest
+```
+
+#### Customize environment
+
+```bash
+cp .env.template .env
+```
+
+And setup env vars according to your needs.
+
+#### Install dependencies
+
+```bash
+go mod download
+```
+
+#### Running
+
+```bash
+go run main.go
+```
+
+## Containerized setup
+
+### Clone the project
+
+### Go to the project directory
+
+```bash
+cd AdNova/services/loadtest
+```
+
+### Build docker image
+
+```bash
+docker build -t adnova-loadtest .
+```
+
+### Customize environment
+
+Customize environment with `docker run` command, for all environment vars and default values see [.env.template](./.env.template).
+
+### Run docker image
+
+```bash
+docker run -p 5001:5001 --name adnova-loadtest adnova-loadtest
+```
+
+Loadtest will be available on [127.0.0.1:5001](http://127.0.0.1:5001).

services/loadtest/go.mod

@@ -0,0 +1,8 @@
+module loadtest
+
+go 1.24.5
+
+require (
+	github.com/gorilla/mux v1.8.1
+	github.com/gorilla/websocket v1.5.3
+)

services/loadtest/go.sum

@@ -0,0 +1,4 @@
+github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
+github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
+github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
+github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=

services/loadtest/main.go

@@ -0,0 +1,788 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"html/template"
+	"io"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/gorilla/mux"
+	"github.com/gorilla/websocket"
+)
+
+// --- Mock Data Structures ---
+
+type Client struct {
+	ClientID string `json:"client_id"`
+	Login    string `json:"login"`
+	Age      int    `json:"age"`
+	Location string `json:"location"`
+	Gender   string `json:"gender"`
+}
+
+type Advertiser struct {
+	AdvertiserID string `json:"advertiser_id"`
+	Name         string `json:"name"`
+}
+
+type MLScore struct {
+	AdvertiserId string `json:"advertiser_id"`
+	ClientID     string `json:"client_id"`
+	Score        int64  `json:"score"`
+}
+
+type Campaign struct {
+	Targeting struct {
+		Gender   string `json:"gender,omitempty"`
+		AgeFrom  int    `json:"age_from,omitempty"`
+		AgeTo    int    `json:"age_to,omitempty"`
+		Location string `json:"location,omitempty"`
+	} `json:"targeting"`
+	AdTitle           string  `json:"ad_title"`
+	AdText            string  `json:"ad_text"`
+	ImpressionsLimit  int     `json:"impressions_limit"`
+	ClicksLimit       int     `json:"clicks_limit"`
+	CostPerImpression float64 `json:"cost_per_impression"`
+	CostPerClick      float64 `json:"cost_per_click"`
+	StartDate         int64   `json:"start_date"`
+	EndDate           int64   `json:"end_date"`
+}
+
+type CampaignFileEntry struct {
+	AdvertiserID string   `json:"advertiser_id"`
+	CampaignData Campaign `json:"campaign_data"`
+}
+
+// --- Mock data filenames ---
+
+const clientsMockDataFile = "./mocks/bulk_clients.json"
+const advertisersMockDataFile = "./mocks/bulk_advertisers.json"
+const campaignsMockDataFile = "./mocks/campaigns.json"
+const mlscoresMockDataFile = "./mocks/ml_scores.json"
+
+// --- In-Memory Data Store ---
+
+var (
+	loadedClientIDs []string
+	dataMutex       = &sync.RWMutex{}
+	backendAddress  string
+)
+
+// --- Mock Data Loading and Posting Logic ---
+
+func loadInitialMockIDs() {
+	dataMutex.Lock()
+	defer dataMutex.Unlock()
+
+	loadedClientIDs = nil
+
+	log.Println("Loading client IDs from local files...")
+
+	readFile := func(filename string) ([]byte, error) {
+		content, err := os.ReadFile(filename)
+		if err != nil {
+			return nil, fmt.Errorf("error reading %s: %w", filename, err)
+		}
+		return content, nil
+	}
+
+	clientsContent, err := readFile(clientsMockDataFile)
+	if err != nil {
+		log.Printf("Warning: Could not read %s for initial ID load: %v", clientsMockDataFile, err)
+	} else {
+		var tempClients []Client
+		if err := json.Unmarshal(clientsContent, &tempClients); err != nil {
+			log.Printf("Warning: Error unmarshaling %s for initial ID load: %v", clientsMockDataFile, err)
+		} else {
+			for _, c := range tempClients {
+				loadedClientIDs = append(loadedClientIDs, c.ClientID)
+			}
+			log.Printf("Loaded %d client IDs for stress testing.", len(loadedClientIDs))
+		}
+	}
+}
+
+func loadAndPostMocks() error {
+	log.Println("Attempting to load mock data from files and post to external backend:", backendAddress)
+
+	readFile := func(filename string) ([]byte, error) {
+		content, err := os.ReadFile(filename)
+		if err != nil {
+			return nil, fmt.Errorf("error reading %s: %w", filename, err)
+		}
+		return content, nil
+	}
+
+	postJSON := func(url string, data interface{}) error {
+		jsonData, err := json.Marshal(data)
+		if err != nil {
+			return fmt.Errorf("failed to marshal JSON for %s: %w", url, err)
+		}
+		req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
+		if err != nil {
+			return fmt.Errorf("failed to create request for %s: %w", url, err)
+		}
+		req.Header.Set("Content-Type", "application/json")
+
+		client := &http.Client{
+			Timeout: 60 * time.Second,
+		}
+
+		resp, err := client.Do(req)
+		if err != nil {
+			return fmt.Errorf("failed to POST to %s: %w", url, err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode >= 400 {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("POST to %s failed with status %d: %s", url, resp.StatusCode, string(body))
+		}
+		return nil
+	}
+
+	// 1. Load bulk_clients.json
+	clientsContent, err := readFile(clientsMockDataFile)
+	if err != nil {
+		return err
+	}
+	var tempClients []Client
+	if err := json.Unmarshal(clientsContent, &tempClients); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", clientsMockDataFile, err)
+	}
+	if err := postJSON(fmt.Sprintf("%s/clients/bulk", backendAddress), tempClients); err != nil {
+		return fmt.Errorf("error posting bulk clients: %w", err)
+	}
+	log.Printf("Successfully posted %d clients to %s/clients/bulk", len(tempClients), backendAddress)
+
+	// 2. Load bulk_advertisers.json
+	advertisersContent, err := readFile(advertisersMockDataFile)
+	if err != nil {
+		return err
+	}
+	var tempAdvertisers []Advertiser
+	if err := json.Unmarshal(advertisersContent, &tempAdvertisers); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", advertisersMockDataFile, err)
+	}
+	if err := postJSON(fmt.Sprintf("%s/advertisers/bulk", backendAddress), tempAdvertisers); err != nil {
+		return fmt.Errorf("error posting bulk advertisers: %w", err)
+	}
+	log.Printf("Successfully posted %d advertisers to %s/advertisers/bulk", len(tempAdvertisers), backendAddress)
+
+	// 3. Load campaigns.json
+	campaignsContent, err := readFile(campaignsMockDataFile)
+	if err != nil {
+		return err
+	}
+	var campaignDataList []CampaignFileEntry
+	if err := json.Unmarshal(campaignsContent, &campaignDataList); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", campaignsMockDataFile, err)
+	}
+
+	for i, entry := range campaignDataList {
+		if err := postJSON(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, entry.AdvertiserID), entry.CampaignData); err != nil {
+			log.Printf("Warning: Failed to post campaign for advertiser %s (entry %d): %v", entry.AdvertiserID, i, err)
+		}
+	}
+	log.Printf("Attempted to post campaigns for %d advertisers from %s", len(campaignDataList), campaignsMockDataFile)
+
+	// 4. Load ml_scores.json
+	mlScoresContent, err := readFile(mlscoresMockDataFile)
+	if err != nil {
+		return err
+	}
+	var tempMLScores []MLScore
+	if err := json.Unmarshal(mlScoresContent, &tempMLScores); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", mlscoresMockDataFile, err)
+	}
+	for i, score := range tempMLScores {
+		if err := postJSON(fmt.Sprintf("%s/ml-scores", backendAddress), score); err != nil {
+			log.Printf("Warning: Failed to post ML score %d (client %s): %v", i, score.ClientID, err)
+		}
+	}
+	log.Printf("Attempted to post %d ML scores one by one to %s/ml-scores", len(tempMLScores), backendAddress)
+
+	log.Println("Mock data loading and posting process complete.")
+	return nil
+}
+
+// --- WebSocket Hub ---
+
+var upgrader = websocket.Upgrader{
+	ReadBufferSize:  1024,
+	WriteBufferSize: 1024,
+	CheckOrigin:     func(r *http.Request) bool { return true },
+}
+
+type Hub struct {
+	clients    map[*websocket.Conn]bool
+	broadcast  chan []byte
+	register   chan *websocket.Conn
+	unregister chan *websocket.Conn
+	mutex      sync.Mutex
+}
+
+func newHub() *Hub {
+	return &Hub{
+		broadcast:  make(chan []byte),
+		register:   make(chan *websocket.Conn),
+		unregister: make(chan *websocket.Conn),
+		clients:    make(map[*websocket.Conn]bool),
+	}
+}
+
+func (h *Hub) run() {
+	for {
+		select {
+		case client := <-h.register:
+			h.mutex.Lock()
+			h.clients[client] = true
+			h.mutex.Unlock()
+		case client := <-h.unregister:
+			h.mutex.Lock()
+			if _, ok := h.clients[client]; ok {
+				delete(h.clients, client)
+				client.Close()
+			}
+			h.mutex.Unlock()
+		case message := <-h.broadcast:
+			h.mutex.Lock()
+			for client := range h.clients {
+				err := client.WriteMessage(websocket.TextMessage, message)
+				if err != nil {
+					log.Printf("error: %v", err)
+					client.Close()
+					delete(h.clients, client)
+				}
+			}
+			h.mutex.Unlock()
+		}
+	}
+}
+
+var hub = newHub()
+
+// --- Load Generator ---
+
+type TestConfig struct {
+	BackendAddress string `json:"backendAddress"`
+	MaxRPS         int    `json:"maxRps"`
+	LoadProfile    string `json:"loadProfile"`
+	FromRPS        int    `json:"fromRPS"`
+	ToRPS          int    `json:"toRPS"`
+	StepRPS        int    `json:"stepRps"`
+	StepDuration   int    `json:"stepDuration"`
+	OnceCount      int    `json:"onceCount"`
+}
+
+type RequestResult struct {
+	StatusCode int
+	Latency    time.Duration
+	Error      bool
+}
+
+type TestStats struct {
+	RPS         int     `json:"rps"`
+	Latency     float64 `json:"latency"`
+	ErrorRate   float64 `json:"errorRate"`
+	TotalReqs   int64   `json:"totalReqs"`
+	TotalErrors int64   `json:"totalErrors"`
+	IsRunning   bool    `json:"isRunning"`
+}
+
+type TestManager struct {
+	config     TestConfig
+	isRunning  bool
+	cancelFunc context.CancelFunc
+	mutex      sync.Mutex
+	httpClient *http.Client
+}
+
+var testManager = TestManager{
+	httpClient: &http.Client{
+		Timeout: 10 * time.Second,
+		Transport: &http.Transport{
+			MaxIdleConns:        1000,
+			MaxIdleConnsPerHost: 1000,
+			IdleConnTimeout:     90 * time.Second,
+		},
+	},
+}
+
+func (tm *TestManager) startTest(config TestConfig) {
+	tm.mutex.Lock()
+	if tm.isRunning {
+		tm.mutex.Unlock()
+		log.Println("Test already running.")
+		return
+	}
+
+	tm.config = config
+	tm.isRunning = true
+	ctx, cancel := context.WithCancel(context.Background())
+	tm.cancelFunc = cancel
+	tm.mutex.Unlock()
+
+	go tm.runLoadGenerator(ctx)
+}
+
+func (tm *TestManager) stopTest() {
+	tm.mutex.Lock()
+	if tm.isRunning && tm.cancelFunc != nil {
+		tm.cancelFunc()
+		tm.isRunning = false
+	}
+	tm.mutex.Unlock()
+}
+
+func (tm *TestManager) runLoadGenerator(ctx context.Context) {
+	log.Printf("Starting test with config: %+v\n", tm.config)
+
+	results := make(chan RequestResult, 10000)
+	var wg sync.WaitGroup
+
+	var totalReqs, totalErrors int64
+	go func() {
+		ticker := time.NewTicker(1 * time.Second)
+		defer ticker.Stop()
+		var reqsInSecond int
+		var totalLatency time.Duration
+		var errorsInSecond int
+
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case <-ticker.C:
+				avgLatency := 0.0
+				if reqsInSecond > 0 {
+					avgLatency = float64(totalLatency.Milliseconds()) / float64(reqsInSecond)
+				}
+				errorRate := 0.0
+				if reqsInSecond > 0 {
+					errorRate = float64(errorsInSecond) / float64(reqsInSecond) * 100
+				}
+
+				stats := TestStats{
+					RPS:         reqsInSecond,
+					Latency:     avgLatency,
+					ErrorRate:   errorRate,
+					TotalReqs:   totalReqs,
+					TotalErrors: totalErrors,
+					IsRunning:   true,
+				}
+				jsonStats, _ := json.Marshal(stats)
+				hub.broadcast <- jsonStats
+
+				reqsInSecond = 0
+				totalLatency = 0
+				errorsInSecond = 0
+			case res, ok := <-results:
+				if !ok {
+					return
+				}
+				totalReqs++
+				reqsInSecond++
+				totalLatency += res.Latency
+				if res.Error {
+					totalErrors++
+					errorsInSecond++
+				}
+			}
+		}
+	}()
+
+	numWorkers := 1000
+	if tm.config.MaxRPS > 0 && tm.config.MaxRPS < numWorkers {
+		numWorkers = tm.config.MaxRPS
+	}
+
+	jobs := make(chan string, 10000)
+
+	for i := 0; i < numWorkers; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				case url, ok := <-jobs:
+					if !ok {
+						return
+					}
+					start := time.Now()
+					req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
+					resp, err := tm.httpClient.Do(req)
+					latency := time.Since(start)
+
+					if err != nil {
+						results <- RequestResult{StatusCode: 0, Latency: latency, Error: true}
+						continue
+					}
+
+					results <- RequestResult{StatusCode: resp.StatusCode, Latency: latency, Error: resp.StatusCode >= 500}
+					resp.Body.Close()
+				}
+			}
+		}()
+	}
+
+	go func() {
+		defer close(jobs)
+
+		dataMutex.RLock()
+		if len(loadedClientIDs) == 0 {
+			log.Println("No client IDs loaded for stress test.")
+			dataMutex.RUnlock()
+			return
+		}
+		clientIDsForTest := make([]string, len(loadedClientIDs))
+		copy(clientIDsForTest, loadedClientIDs)
+		dataMutex.RUnlock()
+
+		log.Printf("Stress testing with %d client IDs.", len(clientIDsForTest))
+
+		switch tm.config.LoadProfile {
+		case "const":
+			ticker := time.NewTicker(time.Second / time.Duration(tm.config.MaxRPS))
+			defer ticker.Stop()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				case <-ticker.C:
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+				}
+			}
+		case "line":
+			duration := 10 * time.Second
+			startTime := time.Now()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				default:
+					elapsed := time.Since(startTime)
+					if elapsed >= duration {
+						elapsed = duration
+					}
+
+					progress := float64(elapsed) / float64(duration)
+					currentRPS := float64(tm.config.FromRPS) + (float64(tm.config.ToRPS-tm.config.FromRPS) * progress)
+
+					if currentRPS <= 0 {
+						currentRPS = 1
+					}
+
+					sleepDuration := time.Second / time.Duration(currentRPS)
+					time.Sleep(sleepDuration)
+
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+
+					if elapsed >= duration {
+						constSleepDuration := time.Second / time.Duration(tm.config.ToRPS)
+						for {
+							select {
+							case <-ctx.Done():
+								return
+							default:
+								time.Sleep(constSleepDuration)
+								clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+								url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+								jobs <- url
+							}
+						}
+					}
+				}
+			}
+		case "step":
+			currentRPS := tm.config.FromRPS
+			for currentRPS <= tm.config.ToRPS {
+				log.Printf("Step load: %d RPS for %d seconds", currentRPS, tm.config.StepDuration)
+				stepEndTime := time.After(time.Duration(tm.config.StepDuration) * time.Second)
+
+				sleepDuration := time.Second / time.Duration(currentRPS)
+
+			stepLoop:
+				for {
+					select {
+					case <-ctx.Done():
+						return
+					case <-stepEndTime:
+						break stepLoop
+					default:
+						time.Sleep(sleepDuration)
+						clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+						url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+						jobs <- url
+					}
+				}
+				currentRPS += tm.config.StepRPS
+				if tm.config.StepRPS == 0 && currentRPS != tm.config.ToRPS {
+					break
+				}
+			}
+		case "once":
+			for i := 0; i < tm.config.OnceCount; i++ {
+				select {
+				case <-ctx.Done():
+					return
+				default:
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+				}
+			}
+		case "unlimited":
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				default:
+					clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
+					url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
+					jobs <- url
+				}
+			}
+		}
+	}()
+
+	wg.Wait()
+	close(results)
+	time.Sleep(1 * time.Second)
+
+	tm.mutex.Lock()
+	tm.isRunning = false
+	tm.mutex.Unlock()
+
+	finalStats := TestStats{IsRunning: false, TotalReqs: totalReqs, TotalErrors: totalErrors}
+	jsonStats, _ := json.Marshal(finalStats)
+	hub.broadcast <- jsonStats
+	log.Println("Test finished.")
+}
+
+// --- HTTP Handlers ---
+
+func serveWs(w http.ResponseWriter, r *http.Request) {
+	conn, err := upgrader.Upgrade(w, r, nil)
+	if err != nil {
+		log.Println(err)
+		return
+	}
+	hub.register <- conn
+
+	go func() {
+		defer func() {
+			hub.unregister <- conn
+			conn.Close()
+		}()
+		for {
+			if _, _, err := conn.NextReader(); err != nil {
+				break
+			}
+		}
+	}()
+}
+
+func handleLoadMocks(w http.ResponseWriter, r *http.Request) {
+	if err := loadAndPostMocks(); err != nil {
+		http.Error(w, fmt.Sprintf("Failed to load mocks: %v", err), http.StatusInternalServerError)
+		return
+	}
+	w.WriteHeader(http.StatusOK)
+	json.NewEncoder(w).Encode(map[string]string{"status": "Mocks loaded successfully"})
+}
+
+func checkEndpoint(url string) bool {
+	req, err := http.NewRequest("GET", url, nil)
+	if err != nil {
+		log.Printf("Error creating request for %s: %v", url, err)
+		return false
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	req = req.WithContext(ctx)
+
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		log.Printf("Error making request to %s: %v", url, err)
+		return false
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		log.Printf("Check for %s failed with status: %d", url, resp.StatusCode)
+		return false
+	}
+
+	if strings.Contains(url, "/campaigns") {
+		bodyBytes, err := io.ReadAll(resp.Body)
+		if err != nil {
+			log.Printf("Error reading response body from %s: %v", url, err)
+			return false
+		}
+		var campaignsData []Campaign
+		if err := json.Unmarshal(bodyBytes, &campaignsData); err != nil {
+			log.Printf("Error unmarshaling campaign data from %s: %v", url, err)
+			return false
+		}
+		return len(campaignsData) > 0
+	}
+	return true
+}
+
+func handleCheckMocks(w http.ResponseWriter, r *http.Request) {
+	dataMutex.RLock()
+	defer dataMutex.RUnlock()
+
+	status := make(map[string]bool)
+
+	var tempClients []Client
+	clientsContent, err := os.ReadFile(clientsMockDataFile)
+	if err != nil {
+		log.Printf("Warning: Could not read %s for checks: %v", clientsMockDataFile, err)
+	} else {
+		json.Unmarshal(clientsContent, &tempClients)
+	}
+
+	var tempAdvertisers []Advertiser
+	advertisersContent, err := os.ReadFile(advertisersMockDataFile)
+	if err != nil {
+		log.Printf("Warning: Could not read %s for checks: %v", advertisersMockDataFile, err)
+	} else {
+		json.Unmarshal(advertisersContent, &tempAdvertisers)
+	}
+
+	clientsLoaded := false
+	if len(tempClients) >= 3 {
+		firstClient := tempClients[0].ClientID
+		medianClient := tempClients[len(tempClients)/2].ClientID
+		lastClient := tempClients[len(tempClients)-1].ClientID
+
+		clientsLoaded = checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, firstClient)) &&
+			checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, medianClient)) &&
+			checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, lastClient))
+	} else if len(tempClients) > 0 {
+		clientsLoaded = checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, tempClients[0].ClientID))
+	}
+	status["clients"] = clientsLoaded
+
+	advertisersLoaded := false
+	if len(tempAdvertisers) >= 3 {
+		firstAdvertiser := tempAdvertisers[0].AdvertiserID
+		medianAdvertiser := tempAdvertisers[len(tempAdvertisers)/2].AdvertiserID
+		lastAdvertiser := tempAdvertisers[len(tempAdvertisers)-1].AdvertiserID
+
+		advertisersLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, firstAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, medianAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, lastAdvertiser))
+	} else if len(tempAdvertisers) > 0 {
+		advertisersLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, tempAdvertisers[0].AdvertiserID))
+	}
+	status["advertisers"] = advertisersLoaded
+
+	campaignsLoaded := false
+	if len(tempAdvertisers) >= 3 {
+		firstAdvertiser := tempAdvertisers[0].AdvertiserID
+		medianAdvertiser := tempAdvertisers[len(tempAdvertisers)/2].AdvertiserID
+		lastAdvertiser := tempAdvertisers[len(tempAdvertisers)-1].AdvertiserID
+
+		campaignsLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, firstAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, medianAdvertiser)) &&
+			checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, lastAdvertiser))
+	} else if len(tempAdvertisers) > 0 {
+		campaignsLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, tempAdvertisers[0].AdvertiserID))
+	}
+	status["campaigns"] = campaignsLoaded
+
+	_, err = os.ReadFile(mlscoresMockDataFile)
+	status["ml_scores"] = err == nil
+	if !status["ml_scores"] {
+		log.Printf("Warning: ML Scores file not found or readable, assuming not loaded for check.")
+	}
+
+	w.Header().Set("Content-Type", "application/json")
+	json.NewEncoder(w).Encode(status)
+}
+
+func handleStartTest(w http.ResponseWriter, r *http.Request) {
+	var config TestConfig
+	if err := json.NewDecoder(r.Body).Decode(&config); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+	config.BackendAddress = backendAddress
+	go testManager.startTest(config)
+	w.WriteHeader(http.StatusOK)
+	json.NewEncoder(w).Encode(map[string]string{"status": "Test started"})
+}
+
+func handleStopTest(w http.ResponseWriter, r *http.Request) {
+	testManager.stopTest()
+	w.WriteHeader(http.StatusOK)
+	json.NewEncoder(w).Encode(map[string]string{"status": "Test stopped"})
+}
+
+func main() {
+	rand.Seed(time.Now().UnixNano())
+
+	port := flag.String("port", "5002", "Port to run the server on")
+	flag.Parse()
+
+	backendAddress = os.Getenv("BACKEND_ADDRESS")
+	if backendAddress == "" {
+		log.Println("BACKEND_ADDRESS environment variable not set. Defaulting to http://localhost:8080")
+		backendAddress = "http://localhost:8080"
+	}
+
+	loadInitialMockIDs()
+
+	go hub.run()
+
+	r := mux.NewRouter()
+
+	r.HandleFunc("/", serveUI)
+	r.HandleFunc("/ws", serveWs)
+	r.HandleFunc("/api/load-mocks", handleLoadMocks).Methods("POST")
+	r.HandleFunc("/api/check-mocks", handleCheckMocks).Methods("GET")
+	r.HandleFunc("/api/start-test", handleStartTest).Methods("POST")
+	r.HandleFunc("/api/stop-test", handleStopTest).Methods("POST")
+
+	addr := ":" + *port
+	log.Printf("Server starting on port %s. Open http://localhost%s\n", *port, addr)
+	if err := http.ListenAndServe(addr, r); err != nil {
+		log.Fatal("ListenAndServe: ", err)
+	}
+}
+
+// --- Embedded HTML UI ---
+
+func serveUI(w http.ResponseWriter, r *http.Request) {
+	w.Header().Set("Content-Type", "text/html; charset=utf-8")
+
+	t, err := template.ParseFiles("static/index.html")
+	if err != nil {
+		http.Error(w, "Could not load UI template", http.StatusInternalServerError)
+		return
+	}
+
+	err = t.Execute(w, nil)
+	if err != nil {
+		http.Error(w, "Failed to render UI", http.StatusInternalServerError)
+	}
+}

services/loadtest/mocks/bulk_advertisers.json

@@ -0,0 +1,122 @@
+[
+    {
+        "advertiser_id": "addf3407-9265-44b8-8e7e-6b015f63c47d",
+        "name": "Smart Inc"
+    },
+    {
+        "advertiser_id": "4d830921-f650-4f2f-869b-db48208dc952",
+        "name": "Advanced Technologies"
+    },
+    {
+        "advertiser_id": "752ddcd3-e944-4eaf-b1de-5e49977354a7",
+        "name": "Global Group"
+    },
+    {
+        "advertiser_id": "e04d35cd-e24b-4bfd-b5fa-6ac1c12ae8a0",
+        "name": "Global Media"
+    },
+    {
+        "advertiser_id": "8b7f547f-bab5-448d-9997-61cbbd6e3f1c",
+        "name": "Modern Systems"
+    },
+    {
+        "advertiser_id": "57dd000f-32fe-4a49-8dca-bfdc84fec7dd",
+        "name": "Advanced Technologies"
+    },
+    {
+        "advertiser_id": "22d28b06-32c5-434e-b05e-0654fa1cac09",
+        "name": "Global Corp"
+    },
+    {
+        "advertiser_id": "7ba54148-4885-48da-97ba-6ecf7b3a300b",
+        "name": "Tech Solutions"
+    },
+    {
+        "advertiser_id": "00ac5657-1a1a-48f1-815a-2ffdb72b82b8",
+        "name": "Elite Group"
+    },
+    {
+        "advertiser_id": "b9d5a8ce-ae23-4a93-946e-4bd85bddef5c",
+        "name": "Modern Marketing"
+    },
+    {
+        "advertiser_id": "3696b94d-6591-4e15-86f3-8c46ee535e27",
+        "name": "Advanced Systems"
+    },
+    {
+        "advertiser_id": "20024aab-2476-4d35-b96f-f47589f05474",
+        "name": "Premier Technologies"
+    },
+    {
+        "advertiser_id": "a68ef7d6-c50f-40d1-b70f-a32d7cdcd036",
+        "name": "Elite Partners"
+    },
+    {
+        "advertiser_id": "960e2896-3f34-43a3-8aa1-d984bb92b96e",
+        "name": "Future Inc"
+    },
+    {
+        "advertiser_id": "80a7f6a9-5939-4d74-9a7e-e455b9fa0fdd",
+        "name": "Modern Marketing"
+    },
+    {
+        "advertiser_id": "6bb59511-360d-49dc-9626-be2f6b9b3a0a",
+        "name": "Future Solutions"
+    },
+    {
+        "advertiser_id": "d7a27dc9-7d8d-4f25-bea4-2528f88238d1",
+        "name": "Advanced Solutions"
+    },
+    {
+        "advertiser_id": "e6cde283-ca4b-456c-b1fb-c2967918b5ef",
+        "name": "Digital Solutions"
+    },
+    {
+        "advertiser_id": "2a51ba38-bed3-43af-bbb4-99093b72e0d1",
+        "name": "Tech Group"
+    },
+    {
+        "advertiser_id": "95a9a488-8fee-4aa6-9c21-458f8f5523ca",
+        "name": "Elite Systems"
+    },
+    {
+        "advertiser_id": "2fb992b8-4441-47f5-9176-1da68526eafb",
+        "name": "Tech Inc"
+    },
+    {
+        "advertiser_id": "f0d160bd-18b4-4c88-917c-06b699abeeec",
+        "name": "Global Inc"
+    },
+    {
+        "advertiser_id": "d8147d8c-2bcd-4ac8-bab5-82e420509a7f",
+        "name": "Smart Inc"
+    },
+    {
+        "advertiser_id": "94209720-2c10-4a55-bc37-c28838d08012",
+        "name": "Advanced Corp"
+    },
+    {
+        "advertiser_id": "33fae4ce-d252-4b34-af2b-c11e770f65b5",
+        "name": "Smart Advertising"
+    },
+    {
+        "advertiser_id": "2c0261ba-bd3c-4b19-a53c-44efa10819e5",
+        "name": "Digital Marketing"
+    },
+    {
+        "advertiser_id": "0892fbb7-fbc6-45e9-b52f-494258a9c8f8",
+        "name": "Smart Group"
+    },
+    {
+        "advertiser_id": "055af498-567a-4a4e-b25f-60a57b9b591c",
+        "name": "Advanced Systems"
+    },
+    {
+        "advertiser_id": "0dfd7246-4cfa-4721-899b-e6dd9c483626",
+        "name": "Premier Advertising"
+    },
+    {
+        "advertiser_id": "32d75bc6-0779-4081-98b3-58273497c561",
+        "name": "Modern Partners"
+    }
+]

services/loadtest/mocks/campaigns.json

@@ -0,0 +1,572 @@
+[
+    {
+        "advertiser_id": "addf3407-9265-44b8-8e7e-6b015f63c47d",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.5,
+            "ad_title": "Amazing Offer",
+            "ad_text": "Get great savings now!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": "MALE",
+                "age_from": 25,
+                "age_to": 40,
+                "location": "Chicago"
+            }
+        }
+    },
+    {
+        "advertiser_id": "4d830921-f650-4f2f-869b-db48208dc952",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.8,
+            "ad_title": "Incredible Deal",
+            "ad_text": "Discover amazing quality!",
+            "start_date": 0,
+            "end_date": 20,
+            "targeting": {
+                "gender": "FEMALE",
+                "age_from": 30,
+                "age_to": 50,
+                "location": "New York"
+            }
+        }
+    },
+    {
+        "advertiser_id": "752ddcd3-e944-4eaf-b1de-5e49977354a7",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.8,
+            "cost_per_click": 2.2,
+            "ad_title": "Best Service",
+            "ad_text": "Buy unique advantages today!",
+            "start_date": 0,
+            "end_date": 22,
+            "targeting": {
+                "gender": "ALL",
+                "age_from": 18,
+                "age_to": 60,
+                "location": "Los Angeles"
+            }
+        }
+    },
+    {
+        "advertiser_id": "e04d35cd-e24b-4bfd-b5fa-6ac1c12ae8a0",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.1,
+            "cost_per_click": 2.9,
+            "ad_title": "Exclusive Opportunity",
+            "ad_text": "Try our new product now!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "8b7f547f-bab5-448d-9997-61cbbd6e3f1c",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.4,
+            "ad_title": "Special Features",
+            "ad_text": "Experience our service today!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "57dd000f-32fe-4a49-8dca-bfdc84fec7dd",
+        "campaign_data": {
+            "impressions_limit": 5,
+            "clicks_limit": 5,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.1,
+            "ad_title": "Limited Time Offer",
+            "ad_text": "Get it before it's gone!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "22d28b06-32c5-434e-b05e-0654fa1cac09",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.7,
+            "cost_per_click": 2.6,
+            "ad_title": "New Arrival",
+            "ad_text": "Check out our latest product!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "7ba54148-4885-48da-97ba-6ecf7b3a300b",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.3,
+            "ad_title": "Discover More",
+            "ad_text": "Find out what we offer!",
+            "start_date": 0,
+            "end_date": 16,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "00ac5657-1a1a-48f1-815a-2ffdb72b82b8",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.4,
+            "cost_per_click": 2.8,
+            "ad_title": "Best Quality",
+            "ad_text": "Experience the best quality!",
+            "start_date": 0,
+            "end_date": 18,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "b9d5a8ce-ae23-4a93-946e-4bd85bddef5c",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.2,
+            "ad_title": "Limited Edition",
+            "ad_text": "Get it while it lasts!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "3696b94d-6591-4e15-86f3-8c46ee535e27",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Exclusive Access",
+            "ad_text": "Join us for exclusive access!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "20024aab-2476-4d35-b96f-f47589f05474",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.8,
+            "cost_per_click": 2.5,
+            "ad_title": "Best Experience",
+            "ad_text": "Experience the best with us!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "a68ef7d6-c50f-40d1-b70f-a32d7cdcd036",
+        "campaign_data": {
+            "impressions_limit": 5,
+            "clicks_limit": 5,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.7,
+            "ad_title": "Incredible Quality",
+            "ad_text": "Get the quality you deserve!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "960e2896-3f34-43a3-8aa1-d984bb92b96e",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.4,
+            "ad_title": "Best Value",
+            "ad_text": "Get the best value for your money!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "80a7f6a9-5939-4d74-9a7e-e455b9fa0fdd",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.6,
+            "ad_title": "Exclusive Offer",
+            "ad_text": "Don't miss out on this offer!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "6bb59511-360d-49dc-9626-be2f6b9b3a0a",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.7,
+            "cost_per_click": 2.3,
+            "ad_title": "Limited Time Deal",
+            "ad_text": "Act fast to get this deal!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "d7a27dc9-7d8d-4f25-bea4-2528f88238d1",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Discover Our Services",
+            "ad_text": "Find out what we can do for you!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "e6cde283-ca4b-456c-b1fb-c2967918b5ef",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.8,
+            "ad_title": "Get Started Today",
+            "ad_text": "Join us and start saving!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "2a51ba38-bed3-43af-bbb4-99093b72e0d1",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.8,
+            "cost_per_click": 2.5,
+            "ad_title": "Best Choice",
+            "ad_text": "Choose the best for your needs!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "95a9a488-8fee-4aa6-9c21-458f8f5523ca",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.2,
+            "ad_title": "Exclusive Access",
+            "ad_text": "Get exclusive access to our services!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "2fb992b8-4441-47f5-9176-1da68526eafb",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.6,
+            "ad_title": "Best Quality",
+            "ad_text": "Experience the best quality!",
+            "start_date": 0,
+            "end_date": 15,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "f0d160bd-18b4-4c88-917c-06b699abeeec",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.4,
+            "cost_per_click": 2.3,
+            "ad_title": "Incredible Quality",
+            "ad_text": "Get the quality you deserve!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "d8147d8c-2bcd-4ac8-bab5-82e420509a7f",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.8,
+            "ad_title": "Discover Our Services",
+            "ad_text": "Find out what we can do for you!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "94209720-2c10-4a55-bc37-c28838d08012",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Best Value",
+            "ad_text": "Get the best value for your money!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "33fae4ce-d252-4b34-af2b-c11e770f65b5",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.4,
+            "ad_title": "Exclusive Offer",
+            "ad_text": "Don't miss out on this offer!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "2c0261ba-bd3c-4b19-a53c-44efa10819e5",
+        "campaign_data": {
+            "impressions_limit": 8,
+            "clicks_limit": 8,
+            "cost_per_impression": 1.9,
+            "cost_per_click": 2.6,
+            "ad_title": "Best Choice",
+            "ad_text": "Choose the best for your needs!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "0892fbb7-fbc6-45e9-b52f-494258a9c8f8",
+        "campaign_data": {
+            "impressions_limit": 9,
+            "clicks_limit": 9,
+            "cost_per_impression": 1.5,
+            "cost_per_click": 2.3,
+            "ad_title": "Incredible Quality",
+            "ad_text": "Get the quality you deserve!",
+            "start_date": 0,
+            "end_date": 14,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "055af498-567a-4a4e-b25f-60a57b9b591c",
+        "campaign_data": {
+            "impressions_limit": 10,
+            "clicks_limit": 10,
+            "cost_per_impression": 1.6,
+            "cost_per_click": 2.8,
+            "ad_title": "Get Started Today",
+            "ad_text": "Join us and start saving!",
+            "start_date": 0,
+            "end_date": 12,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "0dfd7246-4cfa-4721-899b-e6dd9c483626",
+        "campaign_data": {
+            "impressions_limit": 6,
+            "clicks_limit": 6,
+            "cost_per_impression": 1.2,
+            "cost_per_click": 2.9,
+            "ad_title": "Best Experience",
+            "ad_text": "Experience the best with us!",
+            "start_date": 0,
+            "end_date": 11,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    },
+    {
+        "advertiser_id": "32d75bc6-0779-4081-98b3-58273497c561",
+        "campaign_data": {
+            "impressions_limit": 7,
+            "clicks_limit": 7,
+            "cost_per_impression": 1.3,
+            "cost_per_click": 2.2,
+            "ad_title": "Discover Our Services",
+            "ad_text": "Find out what we can do for you!",
+            "start_date": 0,
+            "end_date": 13,
+            "targeting": {
+                "gender": null,
+                "age_from": null,
+                "age_to": null,
+                "location": null
+            }
+        }
+    }
+]

services/loadtest/static/index.html

@@ -0,0 +1,459 @@
+<!DOCTYPE html>
+<html lang="en" class="dark">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>AdNova Loadtest</title>
+    <script src="https://cdn.tailwindcss.com"></script>
+    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
+    <style>
+        body {
+            font-family: 'Inter', sans-serif;
+        }
+
+        @import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');
+
+        .chart-container {
+            position: relative;
+            height: 250px;
+            width: 100%;
+        }
+
+        .status-dot {
+            height: 10px;
+            width: 10px;
+            border-radius: 50%;
+            display: inline-block;
+        }
+
+        .tooltip {
+            visibility: hidden;
+        }
+
+        .has-tooltip:hover .tooltip {
+            visibility: visible;
+        }
+    </style>
+</head>
+
+<body class="bg-slate-900 text-gray-300">
+    <div class="container mx-auto p-4 md:p-8">
+        <header class="mb-8">
+            <h1 class="text-4xl font-bold text-white tracking-tight">AdNova Loadtest</h1>
+        </header>
+
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-8">
+            <!-- Left Column: Controls -->
+            <div class="lg:col-span-1 flex flex-col gap-6">
+
+                <!-- Mock Data Management -->
+                <div class="bg-slate-800 rounded-lg p-6 shadow-lg">
+                    <h2 class="text-xl font-semibold text-white mb-4">1. Data Setup</h2>
+                    <button id="loadMocksBtn"
+                        class="w-full bg-indigo-600 hover:bg-indigo-500 text-white font-bold py-2 px-4 rounded-lg transition-colors duration-300 flex items-center justify-center">
+                        Load Mock Data
+                    </button>
+                    <div id="mockStatus" class="mt-4 space-y-2 text-sm">
+                        <div class="flex items-center justify-between"><span class="text-slate-300">Clients:</span><span
+                                id="clientsStatus" class="font-mono">Checking...</span></div>
+                        <div class="flex items-center justify-between"><span
+                                class="text-slate-300">Advertisers:</span><span id="advertisersStatus"
+                                class="font-mono">Checking...</span></div>
+                        <div class="flex items-center justify-between"><span
+                                class="text-slate-300">Campaigns:</span><span id="campaignsStatus"
+                                class="font-mono">Checking...</span></div>
+                        <div class="flex items-center justify-between"><span class="text-slate-300">ML
+                                Scores:</span><span id="mlScoresStatus" class="font-mono">Checking...</span></div>
+                    </div>
+                </div>
+
+                <!-- Test Configuration -->
+                <div id="config-panel" class="bg-slate-800 rounded-lg p-6 shadow-lg">
+                    <h2 class="text-xl font-semibold text-white mb-4">2. Test Configuration</h2>
+                    <div class="space-y-4">
+                        <div>
+                            <label for="loadProfile" class="block text-sm font-medium text-slate-300">Load
+                                Profile</label>
+                            <select id="loadProfile"
+                                class="mt-1 block w-full bg-slate-700 border border-slate-600 rounded-md shadow-sm py-2 px-3 text-white focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm">
+                                <option value="const">Constant</option>
+                                <option value="line">Line</option>
+                                <option value="step">Step</option>
+                                <option value="once">Once</option>
+                                <option value="unlimited">Unlimited</option>
+                            </select>
+                        </div>
+
+                        <!-- Profile Specific Options -->
+                        <div id="const-options">
+                            <label for="maxRps" class="block text-sm font-medium text-slate-300">Requests Per Second
+                                (RPS)</label>
+                            <input type="number" id="maxRps" value="100"
+                                class="mt-1 block w-full bg-slate-700 border border-slate-600 rounded-md shadow-sm py-2 px-3 text-white focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm">
+                        </div>
+                        <div id="line-options" class="hidden space-y-2">
+                            <input type="number" id="fromRps" placeholder="From RPS (e.g., 10)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="toRps" placeholder="To RPS (e.g., 100)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                        </div>
+                        <div id="step-options" class="hidden space-y-2">
+                            <input type="number" id="stepFromRps" placeholder="From RPS"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="stepToRps" placeholder="To RPS"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="stepRps" placeholder="Step Size (RPS)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                            <input type="number" id="stepDuration" placeholder="Step Duration (sec)"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                        </div>
+                        <div id="once-options" class="hidden">
+                            <input type="number" id="onceCount" placeholder="Number of Requests"
+                                class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
+                        </div>
+                        <div id="unlimited-options"
+                            class="hidden p-2 bg-yellow-900/50 rounded-md text-yellow-300 text-xs">
+                            Warning: This profile sends requests as fast as possible and may exhaust system resources.
+                        </div>
+
+                    </div>
+                </div>
+
+                <!-- Start/Stop Controls -->
+                <div class="bg-slate-800 rounded-lg p-6 shadow-lg">
+                    <h2 class="text-xl font-semibold text-white mb-4">3. Execute Test</h2>
+                    <button id="startStopBtn"
+                        class="w-full bg-green-600 hover:bg-green-500 text-white font-bold py-3 px-4 rounded-lg transition-colors duration-300 text-lg flex items-center justify-center">
+                        Start Test
+                    </button>
+                </div>
+            </div>
+
+            <!-- Right Column: Stats & Charts -->
+            <div class="lg:col-span-2 flex flex-col gap-6">
+                <!-- Summary Stats -->
+                <div class="grid grid-cols-1 md:grid-cols-3 gap-4">
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">RPS</h3>
+                        <p id="rpsStat" class="text-3xl font-semibold text-white">0</p>
+                    </div>
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Avg Latency (ms)</h3>
+                        <p id="latencyStat" class="text-3xl font-semibold text-white">0.00</p>
+                    </div>
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Error Rate</h3>
+                        <p id="errorRateStat" class="text-3xl font-semibold text-white">0.00%</p>
+                    </div>
+                </div>
+                <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Total Requests</h3>
+                        <p id="totalReqsStat" class="text-2xl font-semibold text-white">0</p>
+                    </div>
+                    <div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
+                        <h3 class="text-sm font-medium text-slate-400">Total Errors</h3>
+                        <p id="totalErrorsStat" class="text-2xl font-semibold text-white">0</p>
+                    </div>
+                </div>
+
+                <!-- Charts -->
+                <div class="bg-slate-800 p-4 rounded-lg shadow-lg">
+                    <h3 class="font-semibold text-white mb-2">Requests Per Second (RPS)</h3>
+                    <div class="chart-container"><canvas id="rpsChart"></canvas></div>
+                </div>
+                <div class="bg-slate-800 p-4 rounded-lg shadow-lg">
+                    <h3 class="font-semibold text-white mb-2">Average Latency (ms)</h3>
+                    <div class="chart-container"><canvas id="latencyChart"></canvas></div>
+                </div>
+                <div class="bg-slate-800 p-4 rounded-lg shadow-lg">
+                    <h3 class="font-semibold text-white mb-2">Error Rate (%)</h3>
+                    <div class="chart-container"><canvas id="errorRateChart"></canvas></div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+    <script>
+        const loadProfileSelect = document.getElementById( 'loadProfile' )
+        const startStopBtn = document.getElementById( 'startStopBtn' )
+        const loadMocksBtn = document.getElementById( 'loadMocksBtn' )
+        const configPanel = document.getElementById( 'config-panel' )
+
+        const rpsStat = document.getElementById( 'rpsStat' )
+        const latencyStat = document.getElementById( 'latencyStat' )
+        const errorRateStat = document.getElementById( 'errorRateStat' )
+        const totalReqsStat = document.getElementById( 'totalReqsStat' )
+        const totalErrorsStat = document.getElementById( 'totalErrorsStat' )
+
+        let ws
+        let rpsChart, latencyChart, errorRateChart
+        let isRunning = false
+        const MAX_DATA_POINTS = 60
+
+        function createChart ( ctx, label, color )
+        {
+            return new Chart( ctx, {
+                type: 'line',
+                data: {
+                    labels: [],
+                    datasets: [ {
+                        label: label,
+                        data: [],
+                        borderColor: color,
+                        backgroundColor: color + '33',
+                        borderWidth: 2,
+                        pointRadius: 0,
+                        tension: 0.4,
+                        fill: true,
+                    } ]
+                },
+                options: {
+                    responsive: true,
+                    maintainAspectRatio: false,
+                    scales: {
+                        x: {
+                            ticks: { color: '#94a3b8' },
+                            grid: { color: '#334155' }
+                        },
+                        y: {
+                            beginAtZero: true,
+                            ticks: { color: '#94a3b8' },
+                            grid: { color: '#334155' }
+                        }
+                    },
+                    plugins: {
+                        legend: { display: false }
+                    }
+                }
+            } )
+        }
+
+        function updateChart ( chart, label, newData )
+        {
+            chart.data.labels.push( label )
+            chart.data.datasets[ 0 ].data.push( newData )
+            if ( chart.data.labels.length > MAX_DATA_POINTS )
+            {
+                chart.data.labels.shift()
+                chart.data.datasets[ 0 ].data.shift()
+            }
+            chart.update( 'none' )
+        }
+
+        function resetCharts ()
+        {
+            const charts = [ rpsChart, latencyChart, errorRateChart ]
+            charts.forEach( chart =>
+            {
+                if ( chart )
+                {
+                    chart.data.labels = []
+                    chart.data.datasets[ 0 ].data = []
+                    chart.update()
+                }
+            } )
+            rpsStat.textContent = '0'
+            latencyStat.textContent = '0.00'
+            errorRateStat.textContent = '0.00%'
+            totalReqsStat.textContent = '0'
+            totalErrorsStat.textContent = '0'
+        }
+
+        function setupWebSocket ()
+        {
+            const protocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://'
+            const wsUrl = protocol + window.location.host + '/ws'
+            ws = new WebSocket( wsUrl )
+
+            ws.onopen = () => console.log( 'WebSocket connected' )
+            ws.onclose = () => console.log( 'WebSocket disconnected' )
+            ws.onerror = ( error ) => console.error( 'WebSocket error:', error )
+
+            ws.onmessage = ( event ) =>
+            {
+                const data = JSON.parse( event.data )
+
+                if ( typeof data.isRunning !== 'undefined' )
+                {
+                    if ( data.isRunning )
+                    {
+                        isRunning = true
+                        startStopBtn.textContent = 'Stop Test'
+                        startStopBtn.classList.remove( 'bg-green-600', 'hover:bg-green-500' )
+                        startStopBtn.classList.add( 'bg-red-600', 'hover:bg-red-500' )
+                        configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = true )
+                    }
+                }
+
+                if ( data.isRunning )
+                {
+                    const now = new Date().toLocaleTimeString()
+                    rpsStat.textContent = data.rps
+                    latencyStat.textContent = data.latency.toFixed( 2 )
+                    errorRateStat.textContent = data.errorRate.toFixed( 2 ) + '%'
+                    totalReqsStat.textContent = data.totalReqs
+                    totalErrorsStat.textContent = data.totalErrors
+
+                    updateChart( rpsChart, now, data.rps )
+                    updateChart( latencyChart, now, data.latency )
+                    updateChart( errorRateChart, now, data.errorRate )
+                } else
+                {
+                    totalReqsStat.textContent = data.totalReqs
+                    totalErrorsStat.textContent = data.totalErrors
+                }
+            }
+        }
+
+        function setRunningState ()
+        {
+            isRunning = true
+            startStopBtn.textContent = 'Stop Test'
+            startStopBtn.classList.remove( 'bg-green-600', 'hover:bg-green-500' )
+            startStopBtn.classList.add( 'bg-red-600', 'hover:bg-red-500' )
+            configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = true )
+            resetCharts()
+        }
+
+        function setStoppedState ()
+        {
+            isRunning = false
+            startStopBtn.textContent = 'Start Test'
+            startStopBtn.classList.remove( 'bg-red-600', 'hover:bg-red-500' )
+            startStopBtn.classList.add( 'bg-green-600', 'hover:bg-green-500' )
+            configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = false )
+        }
+
+        startStopBtn.addEventListener( 'click', () =>
+        {
+            if ( isRunning )
+            {
+                fetch( '/api/stop-test', { method: 'POST' } )
+                    .then( () => setStoppedState() )
+            } else
+            {
+                const config = {
+                    loadProfile: loadProfileSelect.value,
+                    maxRps: parseInt( document.getElementById( 'maxRps' ).value ) || 100,
+                    fromRps: parseInt( document.getElementById( 'fromRps' ).value ) || 0,
+                    toRps: parseInt( document.getElementById( 'toRps' ).value ) || 0,
+                    stepFromRps: parseInt( document.getElementById( 'stepFromRps' ).value ) || 0,
+                    stepToRps: parseInt( document.getElementById( 'stepToRps' ).value ) || 0,
+                    stepRps: parseInt( document.getElementById( 'stepRps' ).value ) || 0,
+                    stepDuration: parseInt( document.getElementById( 'stepDuration' ).value ) || 0,
+                    onceCount: parseInt( document.getElementById( 'onceCount' ).value ) || 0,
+                }
+
+                if ( config.loadProfile === 'step' )
+                {
+                    config.fromRps = config.stepFromRps
+                    config.toRps = config.stepToRps
+                }
+
+                fetch( '/api/start-test', {
+                    method: 'POST',
+                    headers: { 'Content-Type': 'application/json' },
+                    body: JSON.stringify( config )
+                } ).then( res =>
+                {
+                    if ( res.ok ) setRunningState()
+                    else alert( "Failed to start test. Check console for details." )
+                } ).catch( err =>
+                {
+                    console.error( "Error starting test:", err )
+                    alert( "Failed to start test. Network error or server unreachable." )
+                } )
+            }
+        } )
+
+        function updateMockStatus ( status )
+        {
+            const statuses = {
+                clients: document.getElementById( 'clientsStatus' ),
+                advertisers: document.getElementById( 'advertisersStatus' ),
+                campaigns: document.getElementById( 'campaignsStatus' ),
+                ml_scores: document.getElementById( 'mlScoresStatus' ),
+            }
+
+            for ( const key in status )
+            {
+                const el = statuses[ key ]
+                if ( el )
+                {
+                    if ( status[ key ] )
+                    {
+                        el.innerHTML = '<span class="status-dot bg-green-500 mr-2"></span>Loaded'
+                        el.classList.remove( 'text-red-400' )
+                        el.classList.add( 'text-green-400' )
+                    } else
+                    {
+                        el.innerHTML = '<span class="status-dot bg-red-500 mr-2"></span>Not Loaded'
+                        el.classList.remove( 'text-green-400' )
+                        el.classList.add( 'text-red-400' )
+                    }
+                }
+            }
+        }
+
+        function checkMocks ()
+        {
+            fetch( '/api/check-mocks' )
+                .then( res => res.json() )
+                .then( data => updateMockStatus( data ) )
+                .catch( err => console.error( "Failed to check mocks:", err ) )
+        }
+
+        loadMocksBtn.addEventListener( 'click', () =>
+        {
+            loadMocksBtn.textContent = 'Loading...'
+            loadMocksBtn.disabled = true
+
+            fetch( '/api/load-mocks', {
+                method: 'POST',
+                headers: { 'Content-Type': 'application/json' },
+                body: JSON.stringify( {} )
+            } )
+                .then( res =>
+                {
+                    if ( !res.ok ) alert( 'Failed to load mocks. Check console for details.' )
+                    return res.json()
+                } )
+                .then( () =>
+                {
+                    checkMocks()
+                } )
+                .finally( () =>
+                {
+                    loadMocksBtn.textContent = 'Load Mock Data'
+                    loadMocksBtn.disabled = false
+                } )
+        } )
+
+        loadProfileSelect.addEventListener( 'change', ( e ) =>
+        {
+            document.getElementById( 'const-options' ).classList.add( 'hidden' )
+            document.getElementById( 'line-options' ).classList.add( 'hidden' )
+            document.getElementById( 'step-options' ).classList.add( 'hidden' )
+            document.getElementById( 'once-options' ).classList.add( 'hidden' )
+            document.getElementById( 'unlimited-options' ).classList.add( 'hidden' )
+
+            document.getElementById( e.target.value + '-options' ).classList.remove( 'hidden' )
+        } )
+
+
+        window.onload = () =>
+        {
+            rpsChart = createChart( document.getElementById( 'rpsChart' ).getContext( '2d' ), 'RPS', '#6366f1' )
+            latencyChart = createChart( document.getElementById( 'latencyChart' ).getContext( '2d' ), 'Latency', '#34d399' )
+            errorRateChart = createChart( document.getElementById( 'errorRateChart' ).getContext( '2d' ), 'Error Rate', '#f87171' )
+
+            setupWebSocket()
+            checkMocks()
+            loadProfileSelect.dispatchEvent( new Event( 'change' ) )
+        };
+    </script>
+</body>
+
+</html>

services/telegram_bot/.env.template

@@ -4,4 +4,4 @@
 AIOGRAM_BOT_TOKEN=
 AIOGRAM_BACKEND_ADDRESS=http://localhost:8080
 REDIS_URI=redis://localhost:6379
-MINIO_ENDPOINT=localhost:9000
+MINIO_ENDPOINT=http://localhost:9000

services/telegram_bot/Dockerfile

@@ -1,7 +1,7 @@
 # Stage 1: Install dependencies
-FROM docker.io/python:3.11-alpine3.20 AS builder
+FROM docker.io/python:3.13-alpine3.22 AS builder
 
-COPY --from=ghcr.io/astral-sh/uv:0.4.30 /uv /uvx /bin/
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
 
 WORKDIR /app
 
@@ -17,7 +17,7 @@ RUN uv sync --no-dev --no-install-project --no-cache
 
 
 # Stage 2: Start the application
-FROM docker.io/python:3.11-alpine3.20
+FROM docker.io/python:3.13-alpine3.22
 
 WORKDIR /app
 
@@ -34,4 +34,4 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONOPTIMIZE=2 \
     PATH="/opt/venv/bin:$PATH"
 
-CMD python main.py
+CMD [ "python", "main.py" ]

services/telegram_bot/README.md

@@ -4,9 +4,9 @@
 
 Ensure you have the following installed on your system:
 
-- [Python](https://www.python.org/) (>=3.10,<3.12)
-- [uv](https://docs.astral.sh/uv/)
-- [Docker](https://www.docker.com/) (for containerized setup)
+- [Python](https://www.python.org/) (>=3.10,<3.14)
+- [uv](https://docs.astral.sh/uv/) (latest version recommended)
+- [Docker](https://www.docker.com/) (for containerized setup, latest version recommended)
 
 ## Basic setup
 

services/telegram_bot/config.py

@@ -13,4 +13,4 @@ API_ENDPOINT = os.getenv("AIOGRAM_BACKEND_URL", "http://localhost:8080")
 
 REDIS_URI = os.getenv("REDIS_URI", "redis://localhost:6379")
 
-MINIO_URL = f"http://{os.getenv('MINIO_ENDPOINT', 'localhost:9000')}"
+MINIO_URL = os.getenv("MINIO_ENDPOINT", "http://localhost:9000")

services/telegram_bot/pyproject.toml

@@ -1,22 +1,19 @@
 [project]
-name = "adnova-telegram_bot"
-version = "0.1.0"
-readme = "README.md"
-requires-python = ">=3.10,<3.12"
 dependencies = [
-    "aiogram-dialog>=2.3.1",
-    "aiogram>=3.17.0",
-    "cachetools>=5.5.1",
-    "httpx>=0.28.1",
-    "openapi-python-client>=0.23.1",
-    "python-dotenv>=1.0.1",
-    "redis>=5.2.1",
+ "aiogram-dialog>=2.4.0,<3.0.0",
+ "aiogram>=3.17.0,<4.0.0",
+ "cachetools>=5.0.0,<6.0.0",
+ "httpx>=0.28.0,<0.29.0",
+ "python-dotenv>=1.1.0,<2.0.0",
+ "redis>=6.2.0,<7.0.0",
 ]
+name = "adnova-telegram_bot"
+readme = "README.md"
+requires-python = ">=3.10,<3.14"
+version = "0.1.0"
 
 [dependency-groups]
-dev = [
-    "ruff>=0.9.6",
-]
+dev = ["ruff"]
 
 [tool.ruff]
 builtins = []
@@ -49,7 +46,7 @@ extend-include = []
 fix = false
 fix-only = false
 force-exclude = true
-include = ["*.py", "*.pyi", "*.ipynb", "**/pyproject.toml"]
+include = ["**/pyproject.toml", "*.ipynb", "*.py", "*.pyi"]
 indent-width = 4
 line-length = 79
 namespace-packages = []
@@ -64,20 +61,20 @@ unsafe-fixes = false
 
 [tool.ruff.analyze]
 detect-string-imports = true
-direction = "Dependencies"
-exclude = []
-include-dependencies = {}
-preview = false
+direction             = "Dependencies"
+exclude               = []
+include-dependencies  = {}
+preview               = false
 
 [tool.ruff.format]
-docstring-code-format = true
+docstring-code-format      = true
 docstring-code-line-length = 79
-exclude = []
-indent-style = "space"
-line-ending = "lf"
-preview = false
-quote-style  = "double"
-skip-magic-trailing-comma = false
+exclude                    = []
+indent-style               = "space"
+line-ending                = "lf"
+preview                    = false
+quote-style                = "double"
+skip-magic-trailing-comma  = false
 
 [tool.ruff.lint]
 allowed-confusables = ["ℹ"]
@@ -92,23 +89,23 @@ extend-unsafe-fixes = []
 external = []
 fixable = ["ALL"]
 ignore = [
-    "ARG",
-    "D",
-    "ANN401",
-    "COM812",
-    "DJ001",
-    "FBT001",
-    "FBT002",
-    "N813",
-    "PLR2004",
-    "RUF001",
-    "TC002",
+ "ANN401",
+ "ARG",
+ "COM812",
+ "D",
+ "DJ001",
+ "FBT001",
+ "FBT002",
+ "N813",
+ "PLR2004",
+ "RUF001",
+ "TC002",
 ]
 logger-objects = []
 per-file-ignores = {}
 preview = false
 select = ["ALL"]
-task-tags = ["TODO", "FIXME", "HACK", "WORKOUT"]
+task-tags = ["FIXME", "HACK", "TODO", "WORKOUT"]
 typing-modules = []
 unfixable = []
 

tests/README.md

@@ -1,6 +1,6 @@
 # AdNova Tests
 
-There is `unit` and `e2e` tests available, unit tests are placed all around `backend` service folder and `e2e` tests placed [here](./e2e/).
+There is `unit` and `e2e` tests available, `unit` tests are placed all around `backend` service folder and `e2e` tests placed [here](./e2e/).
 
 ## Running unit tests
 

tests/e2e/.env.template

@@ -1,3 +1,3 @@
 # Below all environment variables and default values
 
-BACKEND_BASE_URL=http://127.0.0.1:8080
+BACKEND_BASE_URL=http://127.0.0.1:13240

tests/e2e/README.md

@@ -4,14 +4,14 @@
 
 Ensure you have the following installed on your system:
 
-- [Python](https://www.python.org/) (>=3.10,<3.12)
-- [uv](https://docs.astral.sh/uv/)
-- [Docker](https://www.docker.com/)
-- [Docker compose](https://docs.docker.com/compose/) (latest versions)
+- [Python](https://www.python.org/) (>=3.10,<3.14)
+- [uv](https://docs.astral.sh/uv/) (latest version recommended)
+- [Docker](https://www.docker.com/) (latest version recommended)
+- [Docker compose](https://docs.docker.com/compose/) (latest version recommended)
 
 ## Warning
 
-Please note that containers will use ports from 13241 to 13246 and 8080, so there is must be no listeners on this ports range.
+Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
 
 ## Setup
 

tests/e2e/conftest.py

@@ -9,7 +9,7 @@ from dotenv import load_dotenv
 
 load_dotenv()
 
-BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:8080")
+BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:13240")
 
 
 @pytest.fixture(scope="session", autouse=True)
@@ -20,6 +20,16 @@ def docker_compose() -> Generator[None]:
         args=[
             "docker",
             "compose",
+            "--profile",
+            "loadtest",
+            "--profile",
+            "minio",
+            "--profile",
+            "observability",
+            "--profile",
+            "proxy",
+            "--profile",
+            "telegram_bot",
             "down",
         ],
         check=True,
@@ -31,6 +41,8 @@ def docker_compose() -> Generator[None]:
             "compose",
             "--project-name",
             "adnova-testing",
+            "--profile",
+            "minio",
             "up",
             "-d",
             "--build",
@@ -50,6 +62,8 @@ def docker_compose() -> Generator[None]:
             "compose",
             "--project-name",
             "adnova-testing",
+            "--profile",
+            "minio",
             "down",
             "-v",
         ],

tests/e2e/pyproject.toml

@@ -1,18 +1,12 @@
 [project]
-name = "adnova-e2e-tests"
-version = "0.1.0"
-readme = "README.md"
-requires-python = ">=3.10,<3.12"
-dependencies = [
-    "httpx>=0.28.1",
-    "pytest>=8.3.4",
-    "python-dotenv>=1.0.1",
-]
+dependencies    = ["httpx>=0.28.1", "pytest>=8.3.4", "python-dotenv>=1.0.1"]
+name            = "adnova-tests-e2e"
+readme          = "README.md"
+requires-python = ">=3.10,<3.14"
+version         = "0.1.0"
 
 [dependency-groups]
-dev = [
-    "ruff>=0.9.6",
-]
+dev = ["ruff"]
 
 [tool.ruff]
 builtins = []
@@ -45,7 +39,7 @@ extend-include = []
 fix = false
 fix-only = false
 force-exclude = true
-include = ["*.py", "*.pyi", "*.ipynb", "**/pyproject.toml"]
+include = ["**/pyproject.toml", "*.ipynb", "*.py", "*.pyi"]
 indent-width = 4
 line-length = 79
 namespace-packages = []
@@ -60,50 +54,41 @@ unsafe-fixes = false
 
 [tool.ruff.analyze]
 detect-string-imports = true
-direction = "Dependencies"
-exclude = []
-include-dependencies = {}
-preview = false
+direction             = "Dependencies"
+exclude               = []
+include-dependencies  = {}
+preview               = false
 
 [tool.ruff.format]
-docstring-code-format = true
+docstring-code-format      = true
 docstring-code-line-length = 79
-exclude = []
-indent-style = "space"
-line-ending = "lf"
-preview = false
-quote-style  = "double"
-skip-magic-trailing-comma = false
+exclude                    = []
+indent-style               = "space"
+line-ending                = "lf"
+preview                    = false
+quote-style                = "double"
+skip-magic-trailing-comma  = false
 
 [tool.ruff.lint]
-allowed-confusables = ["ℹ"]
-dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
-exclude = ["tests.py"]
-explicit-preview-rules = false
-extend-fixable = []
+allowed-confusables     = ["ℹ"]
+dummy-variable-rgx      = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
+exclude                 = ["tests.py"]
+explicit-preview-rules  = false
+extend-fixable          = []
 extend-per-file-ignores = {}
-extend-safe-fixes = []
-extend-select = []
-extend-unsafe-fixes = []
-external = []
-fixable = ["ALL"]
-ignore = [
-    "ARG",
-    "D",
-    "ANN401",
-    "COM812",
-    "FBT001",
-    "FBT002",
-    "N813",
-    "S101",
-]
-logger-objects = []
-per-file-ignores = {}
-preview = false
-select = ["ALL"]
-task-tags = ["TODO", "FIXME", "HACK", "WORKOUT"]
-typing-modules = []
-unfixable = []
+extend-safe-fixes       = []
+extend-select           = []
+extend-unsafe-fixes     = []
+external                = []
+fixable                 = ["ALL"]
+ignore                  = ["ANN401", "ARG", "COM812", "D", "FBT001", "FBT002", "N813", "S101"]
+logger-objects          = []
+per-file-ignores        = {}
+preview                 = false
+select                  = ["ALL"]
+task-tags               = ["FIXME", "HACK", "TODO", "WORKOUT"]
+typing-modules          = []
+unfixable               = []
 
 [tool.ruff.lint.pylint]
 max-args = 6