PROD.2025/AdNova
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: PROD.2025/AdNova:bc0f7c81df2081429331f4f4dd62d95a916acc2a
Branches Tags
PROD.2025/AdNova:master
...
compare: PROD.2025/AdNova:master
Branches Tags
PROD.2025/AdNova:master

41 Commits
bc0f7c81df ... master

Author SHA1 Message Date
ITQ 55517ae8c9 fix(grafana): fixed false-positive NoData state 2025-07-26 06:02:20 +03:00
ITQ 425c0918e4 chore(admin): actualized resources and credentials 2025-07-26 05:40:26 +03:00
ITQ 7127227350 feat(grafana): added alerts configuration provisioning 2025-07-26 05:34:35 +03:00
ITQ 0db6ed576a feat(grafana): added caddy dashboard 2025-07-26 05:17:27 +03:00
ITQ ada34fd601 chore(grafana): updated root url 2025-07-26 04:37:41 +03:00
ITQ d6b9914c99 feat(caddy): added additional entrypoint for backend 2025-07-26 03:32:24 +03:00
ITQ 9868f5f54a fix(proxy): exposed caddy to all interfaces 2025-07-26 03:07:08 +03:00
ITQ 370852bfe9 feat(grafana): added backend dashboards 
btw, they are incorrect
 2025-07-26 03:03:25 +03:00
ITQ 0472f89a44 fix(prometheus): scrape target 2025-07-26 02:53:56 +03:00
ITQ e1119608ea fix: fixes in caddy monitoring 2025-07-26 02:52:29 +03:00
ITQ 06df05bcdf feat(prometheus): added caddy metrics 2025-07-26 02:29:03 +03:00
ITQ 3591602479 feat(prometheus): added django backend to scrape targets 2025-07-25 21:05:49 +03:00
ITQ b9a17ca31d feat: added redis 2025-07-25 19:00:14 +03:00
ITQ 88b85b3b2e feat: added redis dashboard 2025-07-25 19:00:04 +03:00
ITQ 7fb4145d23 feat: added Celery dashboards 2025-07-25 18:56:29 +03:00
ITQ 2b2dfdfba6 fix(loadtest): fixed browser security error 2025-07-25 17:29:03 +03:00
ITQ e63a9d047b feat(admin): added loadtest to admin 2025-07-25 17:24:43 +03:00
ITQ eccdd27ec5 feat: added more services to proxy 2025-07-25 17:16:14 +03:00
ITQ 2f530f5278 chore(admin): actualized data about resources and credentials 2025-07-25 17:15:49 +03:00
ITQ dc8e1401e0 fix(loadtest): added retry with wss to avoid browser security errors 2025-07-25 17:15:19 +03:00
ITQ 62a233b6c4 feat: added proxy and admin entrypoint 2025-07-25 16:26:06 +03:00
ITQ 8c1ed19966 chore: removed docker compose secrets 2025-07-25 16:21:06 +03:00
ITQ e5f3553f6d chore(docs): improved README layout 2025-07-25 15:43:12 +03:00
ITQ 99dfecc8e8 fix(e2e): added docker compose profiles 2025-07-25 15:27:49 +03:00
ITQ a99cb3d2cc chore(docs): updated ports across the docs 2025-07-25 14:03:41 +03:00
ITQ df0083e334 chore(grafana): improvements to advertiser business analytics dashboard 2025-07-25 13:34:40 +03:00
ITQ a5ec3ca6cb chore(docker_compose): improvements 2025-07-25 13:33:03 +03:00
ITQ b985818f5a chore(grafana): improved dashboards provider config 2025-07-25 13:32:21 +03:00
ITQ b441ea4832 feat(grafana): added provisioning prometheus datasource 2025-07-25 13:31:45 +03:00
ITQ 5b0e0e07a6 feat(docker_compose): added profiles to compose.yaml 
also refactored it a lil bit
 2025-07-25 12:56:41 +03:00
ITQ bb65de3b99 chore: small improvements 2025-07-25 02:08:41 +03:00
ITQ 3804495ce5 feat: added loadtest service 2025-07-25 00:59:54 +03:00
ITQ dd0568bf91 feat: added zipkin and loadtest services to compose.yaml 2025-07-25 00:57:59 +03:00
ITQ 8f5778fd1a feat(backend): added tracing 2025-07-25 00:57:29 +03:00
ITQ 925f820bfd feat(backend): added profiling 2025-07-24 16:57:47 +03:00
ITQ 0eec2f2187 feat(backend): added django_prometheus 2025-07-24 16:55:58 +03:00
ITQ 05029106f6 chore(deps): update images tags in compose 2025-07-22 21:58:06 +03:00
ITQ 8549700752 chore: services code refactor 2025-07-22 21:56:18 +03:00
ITQ f5c9b69b45 feat: added logos 2025-07-22 21:54:27 +03:00
ITQ 0ab9a70645 chore(deps): updated versions 2025-07-22 21:54:10 +03:00
ITQ d1a0f20c49 chore(telegram_bot): small improvements 2025-07-20 00:05:11 +03:00
68 changed files with 46907 additions and 894 deletions
Expand all files Collapse all files
README.md
+13 -9
View File
@@ -32,7 +32,7 @@ Table Report:
#### Warning
Please note that containers will use ports from 13241 to 13245 and 8080, so there is must be no listeners on this ports range.
Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
#### Configure
@@ -66,7 +66,7 @@ docker compose up -d --build
#### Structure
- **backend**: [127.0.0.1:8080](http://127.0.0.1:8080) -> `8080`
- **backend**: [127.0.0.1:13240](http://127.0.0.1:13240) -> `8080`
- Depends on: `postgres`, `redis`, `minio`, `backend-initdb`
- **backend-initdb**
- Depends on: `postgres`, `redis`, `minio`
@@ -109,7 +109,7 @@ aiogram is a modern and fully asynchronous framework for Telegram bot developmen
### [Redis](https://redis.io/)
Redis is an in-memory data structure store often used as a database, cache, and message broker. It supports various data structures and offers high performance for read and write operations, making it suitable for caching and real-time analytics. Very popular and has big community for today. In project used as fsm for aiogram (to avoid data loss on restart), caches (current_date, mlscores, clicks, views) for backend and as broker for Celery.
Redis is an in-memory data structure store often used as a database, cache, and message broker. It supports various data structures and offers high performance for read and write operations, making it suitable for caching and real-time analytics. Very popular and has big community for today. In project used as fsm for aiogram (to avoid data loss on restart), caches (mlscores, clicks, views) for backend and as broker for Celery.
### [Postgres](https://www.postgresql.org/)
@@ -139,7 +139,7 @@ You may say: "For what we need a lot of complex technologies for now". I have an
### Restful API
API Base endpoint when deployed with default docker compose: [127.0.0.1:8080](http://127.0.0.1:8080), also see [docs](#openapi-docs).
API Base endpoint when deployed with default docker compose: [127.0.0.1:13240](http://127.0.0.1:13240), also see [docs](#openapi-docs).
### Admin panel
@@ -152,9 +152,13 @@ Link: [t.me/adnova_bot](https://t.me/adnova_bot)
Basic commands:
`/start` - Start the bot and authenticate as advertiser
`/help` - Get list of all commands
`/campaigns` - Manage advertiser campaigns (only after authentication)
`/statistics` - See advertiser overall statistics (only after authentication)
`/logout` - Logout of current advertiser account (only after authentication)
See [this](#telegram-bot-1).
@@ -234,9 +238,9 @@ Moderation implemented via report system. Client goes to `/report` ([see OpenAPI
Also admin user (whose credentials specified lower) can add new staff members and even create a specified group for them (this is built-in django capabilities).
Report has four states: Sent, Under review, Took action and Skipped. Admin panel has filtration by states and by flagged by llm status.
Admin panel when deployed with docker compose (by default): [localhost:8080/admin/](http://localhost:8080/admin/)
Admin panel when deployed with docker compose (by default): [localhost:13240/admin/](http://localhost:13240/admin/)
Reports list when deployed with docker compose (requires authentication): [localhost:8080/admin/campaign/campaignreport/](http://localhost:8080/admin/campaign/campaignreport/)
Reports list when deployed with docker compose (requires authentication): [localhost:13240/admin/campaign/campaignreport/](http://localhost:13240/admin/campaign/campaignreport/)
Default username: `admin`
@@ -268,7 +272,7 @@ Default login: `admin`
Default password: `proooooood`
Analytics dashboard when deployed with default docker compose: [localhost:13243/d/adnova-statistics/statistics](http://localhost:13243/d/adnova-statisticss/statistics). You can enter advertiser id and get detailed advertiser statistics and also detailed statistics for each advertiser's campaign.
Analytics dashboard when deployed with default docker compose: [localhost:13243/d/adnova-advertiser-statistics](http://localhost:13243/d/adnova-advertiser-statistics/statistics). You can enter advertiser id and get detailed advertiser statistics and also detailed statistics for each advertiser's campaign.
Demonstration:
@@ -276,11 +280,11 @@ Demonstration:
### OpenAPI docs
When deployed with default docker compose: [localhost:8080/docs](http://localhost:8080/docs)
When deployed with default docker compose: [localhost:13240/docs](http://localhost:13240/docs)
### Healthcheck endpoint
When deployed with default docker compose: [localhost:8080/health](http://localhost:8080/health)
When deployed with default docker compose: [localhost:13240/health](http://localhost:13240/health)
Lets developers easily understand and identify problem and users check services health.
assets/logos/adnova-cleanbg.png
LFS
BIN
View File
Binary file not shown.
assets/logos/adnova.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

compose.yaml
+161 -36
View File
@@ -5,6 +5,9 @@ services:
build:
context: ./services/backend
dockerfile: Dockerfile
tags:
- adnova-backend:latest
pull: true
depends_on:
backend-initdb:
restart: false
@@ -18,10 +21,6 @@ services:
restart: false
condition: service_healthy
required: true
minio:
restart: false
condition: service_healthy
required: true
env_file:
- path: ./infrastructure/backend/.env.template
required: true
@@ -30,15 +29,20 @@ services:
ports:
- name: web
target: 8080
published: 8080
published: 13240
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
restart: unless-stopped
shm_size: 4mb
backend-initdb:
build:
context: ./services/backend
dockerfile: Dockerfile
tags:
- adnova-backend:latest
pull: true
command: ./scripts/initdb
depends_on:
postgres:
@@ -49,27 +53,27 @@ services:
restart: false
condition: service_healthy
required: true
minio:
restart: false
condition: service_healthy
required: true
env_file:
- path: ./infrastructure/backend/.env.template
required: true
- path: ./infrastructure/backend/.env
required: false
shm_size: 4mb
backend-staticfiles:
build:
context: ./services/backend
dockerfile: Dockerfile.staticfiles
tags:
- adnova-backend-staticfiles:latest
pull: true
env_file:
- path: ./infrastructure/backend/.env.template
required: true
- path: ./infrastructure/backend/.env
required: false
healthcheck:
test: ["CMD", "service", "nginx", "status", "||", " exit 1"]
test: ["CMD-SHELL", "nginx", "-t", "||", "exit 1"]
interval: 1m30s
timeout: 5s
start_period: 5s
@@ -81,12 +85,17 @@ services:
published: 13241
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
restart: unless-stopped
shm_size: 4mb
backend-celery-worker:
build:
context: ./services/backend
dockerfile: Dockerfile
tags:
- adnova-backend:latest
pull: true
command: celery -A config worker -l INFO
depends_on:
redis:
@@ -106,9 +115,11 @@ services:
start_period: 10s
start_interval: 2s
restart: unless-stopped
shm_size: 4mb
celery-exporter:
image: docker.io/danihodovic/celery-exporter:0.11.1
image: docker.io/danihodovic/celery-exporter:0.12.2
command: --retry-interval=5
depends_on:
redis:
restart: false
@@ -119,7 +130,10 @@ services:
required: true
- path: ./infrastructure/celery-exporter/.env
required: false
profiles:
- observability
restart: unless-stopped
shm_size: 4mb
telegram_bot:
build:
@@ -127,6 +141,7 @@ services:
dockerfile: Dockerfile
tags:
- adnova-telegram_bot:latest
pull: true
depends_on:
backend:
restart: false
@@ -145,7 +160,10 @@ services:
required: true
- path: ./infrastructure/telegram_bot/.env
required: false
profiles:
- telegram_bot
restart: unless-stopped
shm_size: 4mb
redis:
image: docker.io/redis:7-alpine3.21
@@ -171,9 +189,10 @@ services:
- type: volume
source: redis_data
target: /data
read_only: false
redis-exporter:
image: docker.io/oliver006/redis_exporter:v1.67.0-alpine
image: docker.io/oliver006/redis_exporter:v1.74.0-alpine
depends_on:
redis:
restart: false
@@ -184,11 +203,13 @@ services:
required: true
- path: ./infrastructure/redis-exporter/.env
required: false
profiles:
- observability
restart: unless-stopped
shm_size: 4mb
postgres:
image: docker.io/postgres:17-alpine3.21
image: docker.io/postgres:17-alpine3.22
configs:
- source: postgres_config
target: /etc/postgresql/postgresql.conf
@@ -198,7 +219,7 @@ services:
- path: ./infrastructure/postgres/.env
required: false
healthcheck:
test: ["CMD", "pg_isready"]
test: ["CMD", "pg_isready", "-U", "postgres"]
interval: 1m30s
timeout: 5s
start_period: 5s
@@ -206,17 +227,15 @@ services:
retries: 5
oom_kill_disable: true
restart: unless-stopped
secrets:
- source: postgres_password
target: /run/secrets/postgres_password
shm_size: 128mb
volumes:
- type: volume
source: postgres_data
target: /var/lib/postgresql/data
read_only: false
postgres-exporter:
image: quay.io/prometheuscommunity/postgres-exporter:v0.16.0
image: quay.io/prometheuscommunity/postgres-exporter:v0.17.1
depends_on:
postgres:
restart: false
@@ -227,13 +246,15 @@ services:
required: true
- path: ./infrastructure/postgres-exporter/.env
required: false
profiles:
- observability
restart: unless-stopped
shm_size: 4mb
pgadmin:
image: docker.io/dpage/pgadmin4:9
image: docker.io/dpage/pgadmin4:9.6
configs:
- source: pgadmin_servers
- source: pgadmin_servers_config
target: /pgadmin4/servers.json
depends_on:
postgres:
@@ -258,22 +279,23 @@ services:
published: 13242
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
profiles:
- observability
restart: unless-stopped
secrets:
- source: pgadmin_password
target: /run/secrets/pgadmin_password
shm_size: 4mb
volumes:
- type: volume
source: pgadmin_data
target: /var/lib/pgadmin
read_only: false
grafana:
image: docker.io/grafana/grafana-oss:11.5.0
image: docker.io/grafana/grafana-oss:12.0.2
entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
configs:
- source: grafana_config
target: /usr/share/grafana/conf/defaults.ini
entrypoint: ["/etc/grafana/scripts/entrypoint.sh"]
healthcheck:
test: ["CMD", "wget", "-O", "-", "http://localhost:3000/api/health"]
interval: 1m30s
@@ -287,23 +309,28 @@ services:
published: 13243
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
profiles:
- observability
restart: unless-stopped
shm_size: 4mb
volumes:
- type: volume
source: grafana_data
target: /var/lib/grafana
read_only: false
- type: bind
source: ./infrastructure/grafana/provisioning
target: /etc/grafana/provisioning
read_only: true
- type: bind
source: ./infrastructure/grafana/scripts
target: /etc/grafana/scripts
read_only: true
prometheus:
image: docker.io/prom/prometheus:v3.1.0
command:
- "--config.file=/etc/prometheus/prometheus.yaml"
image: docker.io/prom/prometheus:v3.5.0
command: --config.file=/etc/prometheus/prometheus.yaml
configs:
- source: prometheus_config
target: /etc/prometheus/prometheus.yaml
@@ -320,16 +347,20 @@ services:
published: 13244
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
profiles:
- observability
restart: unless-stopped
shm_size: 4mb
volumes:
- type: volume
source: prometheus_data
target: /prometheus
read_only: false
minio:
image: docker.io/minio/minio:RELEASE.2025-07-18T21-56-31Z
command: server --console-address ":9001"
image: docker.io/minio/minio:RELEASE.2025-02-03T21-03-04Z
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 1m30s
@@ -348,17 +379,113 @@ services:
published: 13245
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
- name: console
target: 9001
published: 13246
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
profiles:
- minio
restart: unless-stopped
shm_size: 4mb
volumes:
- type: volume
source: minio_data
target: /data
read_only: false
zipkin:
image: docker.io/openzipkin/zipkin:3
healthcheck:
test: ["CMD", "wget", "-O", "-", "http://localhost:9411/health"]
interval: 1m30s
timeout: 5s
start_period: 5s
start_interval: 2s
retries: 5
ports:
- name: web
target: 9411
published: 13247
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
profiles:
- observability
restart: unless-stopped
shm_size: 4mb
loadtest:
build:
context: ./services/loadtest
dockerfile: Dockerfile
tags:
- adnova-loadtest:latest
depends_on:
backend:
restart: false
condition: service_healthy
required: true
env_file:
- path: ./infrastructure/loadtest/.env.template
required: true
- path: ./infrastructure/loadtest/.env
required: false
ports:
- name: web
target: 5001
published: 13248
host_ip: 127.0.0.1
protocol: tcp
app_protocol: http
profiles:
- loadtest
restart: unless-stopped
shm_size: 4mb
proxy:
image: docker.io/caddy:2-alpine
configs:
- source: caddy_config
target: /etc/caddy/Caddyfile
ports:
- name: http
target: 80
published: 80
host_ip: 0.0.0.0
protocol: tcp
app_protocol: http
- name: https
target: 443
published: 443
host_ip: 0.0.0.0
protocol: tcp
app_protocol: http
- name: http3
target: 443
published: 443
host_ip: 0.0.0.0
protocol: udp
app_protocol: http
profiles:
- proxy
restart: unless-stopped
shm_size: 4mb
volumes:
- type: volume
source: caddy_data
target: /data
read_only: false
- type: volume
source: caddy_config
target: /config
read_only: false
- type: bind
source: ./infrastructure/caddy/static
target: /var/www
read_only: true
volumes:
redis_data:
@@ -367,21 +494,19 @@ volumes:
grafana_data:
prometheus_data:
minio_data:
caddy_data:
caddy_config:
configs:
redis_config:
file: ./infrastructure/redis/redis.conf
postgres_config:
file: ./infrastructure/postgres/postgresql.conf
pgadmin_servers:
pgadmin_servers_config:
file: ./infrastructure/pgadmin/servers.json
grafana_config:
file: ./infrastructure/grafana/grafana.ini
prometheus_config:
file: ./infrastructure/prometheus/prometheus.yaml
secrets:
postgres_password:
file: ./infrastructure/postgres/password
pgadmin_password:
file: ./infrastructure/pgadmin/password
caddy_config:
file: ./infrastructure/caddy/Caddyfile
infrastructure/backend/.env.template
+5 -1
View File
@@ -1,4 +1,4 @@
DJANGO_SECRET_KEY=secretees
DJANGO_SECRET_KEY=very_insecure_key
DJANGO_DEBUG=False
DJANGO_ALLOWED_HOSTS=*
DJANGO_CSRF_TRUSTED_ORIGINS=http://localhost,http://127.0.0.1
@@ -21,3 +21,7 @@ MINIO_ENDPOINT=minio:9000
MINIO_CUSTOM_ENDPOINT_URL=http://127.0.0.1:13244
MINIO_ACCESS_KEY=admin
MINIO_SECRET_KEY=password
OTEL_METRICS_EXPORTER=none
OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://zipkin:9411/api/v2/spans
OTEL_TRACES_EXPORTER=zipkin_json
infrastructure/caddy/Caddyfile
+68
View File
@@ -0,0 +1,68 @@
{
admin :2019
metrics {
per_host
}
}
(basic-auth) {
basic_auth {
admin $2a$14$2zQilpLka2h8Sn1mmOLAAezwDN8Zy8Ta36WECk4qt5MTn3CWksR0m
}
}
adnova.itqdev.xyz {
@healthPath path /health /health/*
handle @healthPath {
import basic-auth
reverse_proxy http://backend:8080
}
handle_path /static/* {
reverse_proxy http://backend-staticfiles:80
}
reverse_proxy http://backend:8080
}
:8080 {
reverse_proxy http://backend:8080
}
admin.adnova.itqdev.xyz {
import basic-auth
root * /var/www/admin
file_server
}
loadtest.adnova.itqdev.xyz {
import basic-auth
reverse_proxy http://loadtest:5001
}
grafana.adnova.itqdev.xyz {
reverse_proxy http://grafana:3000
}
pgadmin.adnova.itqdev.xyz {
reverse_proxy http://pgadmin:80
}
zipkin.adnova.itqdev.xyz {
import basic-auth
reverse_proxy http://zipkin:9411
}
prometheus.adnova.itqdev.xyz {
import basic-auth
reverse_proxy http://prometheus:9090
}
minio.adnova.itqdev.xyz {
reverse_proxy http://minio:9000
}
infrastructure/caddy/static/admin/index.html
+209
View File
@@ -0,0 +1,209 @@
<!DOCTYPE html>
<html lang="en" class="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AdNova Admin Resources</title>
<script src="https://cdn.tailwindcss.com"></script>
<style>
body {
font-family: 'Inter', sans-serif;
}
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');
</style>
</head>
<body class="bg-slate-900 text-gray-300">
<div class="container mx-auto p-4 md:p-8">
<header class="mb-8">
<h1 class="text-4xl font-bold text-white tracking-tight">AdNova Admin Resources Dashboard</h1>
<p class="text-indigo-400">Quick access to essential tools and credentials.</p>
</header>
<div id="resources-container" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
</div>
</div>
<script>
const resources = [
{
category: "Development Tools",
items: [
{
name: "Gitea Repository",
link: "https://git.itqdev.xyz/PROD.2025/AdNova",
description: "Main monorepo on Gitea."
},
{
name: "Rest API Docs",
link: "https://adnova.itqdev.xyz/docs",
description: "Rest API Docs on Swagger."
},
{
name: "Django Admin",
link: "https://adnova.itqdev.xyz/admin/",
username: "admin",
password: "admin",
description: "Django Admin panel."
},
{
name: "Loadtest",
link: "https://loadtest.adnova.itqdev.xyz",
username: "admin",
password: "kit2025_observability",
description: "Stress test utility with multiple profiles and ability to load mocks."
}
]
},
{
category: "Monitoring",
items: [
{
name: "Grafana",
link: "https://grafana.adnova.itqdev.xyz",
username: "admin",
password: "proooooood",
description: "Real-time system metrics and dashboards."
},
{
name: "Prometheus",
link: "https://prometheus.adnova.itqdev.xyz",
username: "admin",
password: "kit2025_observability",
description: "Real-time system metrics and scrape targets."
},
{
name: "Zipkin",
link: "https://zipkin.adnova.itqdev.xyz",
username: "admin",
password: "kit2025_observability",
description: "Latest traces from production."
},
{
name: "Django Silk",
link: "https://adnova.itqdev.xyz/silk/",
username: "admin",
password: "admin",
description: "App profiles and performance info in production, need to authenticate first as staff in Django Admin."
},
{
name: "Adnova Alerts",
link: "https://t.me/adnova_alerts",
description: "Telegram channel where alerts are posted."
},
{
name: "Healthcheck",
link: "https://adnova.itqdev.xyz/health",
username: "admin",
password: "kit2025_observability",
description: "Summary of integration status between services."
},
{
name: "Uptime-Kuma Dashboard",
link: "https://status.adnova.itqdev.xyz",
description: "External monitoring of AdNova."
}
]
},
{
category: "Databases",
items: [
{
name: "Pgadmin",
link: "https://pgadmin.adnova.itqdev.xyz",
username: "admin@mail.com",
password: "password",
description: "Access to production database."
}
]
}
]
function copyToClipboard ( text )
{
const textarea = document.createElement( 'textarea' )
textarea.value = text
document.body.appendChild( textarea )
textarea.select()
document.execCommand( 'copy' )
document.body.removeChild( textarea )
alert( 'Copied to clipboard!' )
}
function renderResources ()
{
const container = document.getElementById( 'resources-container' )
container.innerHTML = ''
resources.forEach( categoryData =>
{
const categorySection = document.createElement( 'div' )
categorySection.className = 'col-span-full mb-4'
categorySection.innerHTML = `
<h2 class="text-2xl font-semibold text-white mb-4 border-b border-slate-700 pb-2">
${ categoryData.category }
</h2>
`
container.appendChild( categorySection )
categoryData.items.forEach( resource =>
{
const resourceCard = document.createElement( 'div' )
resourceCard.className = 'bg-slate-800 rounded-lg p-6 shadow-lg flex flex-col justify-between'
let credentialsHtml = ''
if ( resource.username || resource.password )
{
credentialsHtml = `
<div class="mt-4 pt-4 border-t border-slate-700">
<h4 class="text-sm font-medium text-slate-400 mb-2">Credentials:</h4>
${ resource.username ? `
<div class="flex items-center text-sm mb-2">
<span class="font-semibold text-slate-300 w-24">Username:</span>
<span id="user-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.username }</span>
<button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.username }">Copy</button>
</div>` : '' }
${ resource.password ? `
<div class="flex items-center text-sm">
<span class="font-semibold text-slate-300 w-24">Password:</span>
<span id="pass-${ resource.name.replace( /\s/g, '' ) }" class="flex-grow text-white font-mono mr-2">${ resource.password }</span>
<button class="copy-btn bg-slate-700 hover:bg-slate-600 text-slate-300 text-xs px-2 py-1 rounded-md" data-text="${ resource.password }">Copy</button>
</div>` : '' }
</div>
`
}
resourceCard.innerHTML = `
<div>
<h3 class="text-xl font-semibold text-white mb-2">${ resource.name }</h3>
<p class="text-sm text-slate-400 mb-4">${ resource.description || 'No description provided.' }</p>
</div>
<div class="mt-auto">
<a href="${ resource.link }" target="_blank" class="inline-flex items-center justify-center bg-indigo-600 hover:bg-indigo-500 text-white font-bold py-2 px-4 rounded-lg transition-colors duration-300 text-sm">
Go to Resource
<svg class="ml-2 -mr-1 w-4 h-4" fill="currentColor" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg"><path d="M11 3a1 1 0 100 2h2.586l-6.293 6.293a1 1 0 101.414 1.414L15 6.414V9a1 1 0 102 0V4a1 1 0 00-1-1h-5z"></path><path d="M5 5a2 2 0 00-2 2v8a2 2 0 002 2h8a2 2 0 002-2v-3a1 1 0 10-2 0v3H5V7h3a1 1 0 000-2H5z"></path></svg>
</a>
${ credentialsHtml }
</div>
`
container.appendChild( resourceCard )
} )
} )
document.querySelectorAll( '.copy-btn' ).forEach( button =>
{
button.addEventListener( 'click', ( event ) =>
{
const textToCopy = event.target.dataset.text
copyToClipboard( textToCopy )
} )
} )
}
window.onload = renderResources;
</script>
</body>
</html>
infrastructure/grafana/grafana.ini
+1 -1
View File
@@ -48,7 +48,7 @@ domain = localhost
enforce_domain = false
# The full public facing url
root_url = %(protocol)s://%(domain)s:%(http_port)s/
root_url = https://grafana.adnova.itqdev.xyz
# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
serve_from_sub_path = false
infrastructure/grafana/provisioning/alerting/contact_points.yaml
+17
View File
@@ -0,0 +1,17 @@
apiVersion: 1
contactPoints:
- orgId: 1
name: Telegram
receivers:
- uid: aet1srtyc40lca
type: telegram
settings:
bottoken: 7797967907:AAGZuUzzuS4LLb525rDNY52Awc2tvpsLjd4
chatid: "-1002555823797"
disable_notification: false
disable_web_page_preview: false
message: '{{ template "telegram.default.message" . }}'
parse_mode: Markdown
protect_content: false
disableResolveMessage: false
infrastructure/grafana/provisioning/alerting/rules.yaml
+141
View File
@@ -0,0 +1,141 @@
apiVersion: 1
groups:
- orgId: 1
name: Default
folder: Backend
interval: 10s
rules:
- uid: aet1xbx1yaupsb
title: Backend p99 > 500 ms
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
editorMode: code
expr: |
histogram_quantile(
0.99,
sum(
rate(
caddy_http_request_duration_seconds_bucket{instance="proxy:2019",handler="reverse_proxy",host="proxy:8080",job="caddy"}[$__rate_interval]
)
) by (le)
)
instant: true
intervalMs: 1000
legendFormat: __auto
maxDataPoints: 43200
range: false
refId: A
- refId: C
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 0.5
type: gte
operator:
type: and
query:
params:
- C
reducer:
params: []
type: last
type: query
datasource:
type: __expr__
uid: __expr__
expression: A
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
dashboardUid: e3a78c36-2f34-4ad6-81d5-284002896829
panelId: 32
noDataState: OK
execErrState: Error
for: 10s
keepFiringFor: 10s
annotations:
__dashboardUid__: e3a78c36-2f34-4ad6-81d5-284002896829
__panelId__: "32"
runbook_url: https://admin.adnova.itqdev.xyz
summary: p99 > 500 ms
isPaused: false
notification_settings:
receiver: Telegram
- orgId: 1
name: Default
folder: Postgres
interval: 10s
rules:
- uid: fet1txr4slywwe
title: "> 100 QPS on Postgresql"
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
editorMode: code
expr: |
sum(
irate(pg_stat_database_xact_commit{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
)
+ sum(
irate(pg_stat_database_xact_rollback{datname="postgres",instance="postgres-exporter:9187",job="postgres"}[5m])
)
instant: true
intervalMs: 1000
legendFormat: __auto
maxDataPoints: 43200
range: false
refId: A
- refId: C
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 100
type: gte
operator:
type: and
query:
params:
- C
reducer:
params: []
type: last
type: query
datasource:
type: __expr__
uid: __expr__
expression: A
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
dashboardUid: postgres-overview
panelId: 14
noDataState: NoData
execErrState: Error
for: 10s
keepFiringFor: 1m
annotations:
__dashboardUid__: postgres-overview
__panelId__: "14"
runbook_url: https://admin.adnova.itqdev.xyz
summary: Postgresql QPS exceeded 100
isPaused: false
notification_settings:
receiver: Telegram
infrastructure/grafana/provisioning/alerting/templates.yaml
+54
View File
@@ -0,0 +1,54 @@
apiVersion: 1
templates:
- orgId: 1
name: Telegram
template: |
{{ define "telegram.default.message" }} {{ if gt (len .Alerts.Firing) 0 }}
🔥🚨 *FIRE IN THE HOLE!* 🚨🔥
We've got *{{ len .Alerts.Firing }} firing alert(s)* that need your immediate attention!
{{ range .Alerts.Firing }}
---
*Alert:* `{{ .Labels.alertname }}`
{{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
*Status:* 🔴 *FIRING* since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }}
{{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
{{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
*Labels:*
{{ range .Labels.SortedPairs }} • `{{ .Name }}` = `{{ .Value }}`
{{ end }}
{{ if gt (len .Annotations) 0 }}*Annotations:*
{{ range .Annotations.SortedPairs }} • `{{ .Name }}` = `{{ .Value }}`
{{ end }}{{ end }}
{{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
{{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
{{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }})
{{ end }}{{ if .SilenceURL }}🤫 [Silence Alert]({{ .SilenceURL }})
{{ end }}--- {{ end }} {{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
✅🟢 *ALL CLEAR!* 🟢✅
Great news! *{{ len .Alerts.Resolved }} alert(s)* have been resolved.
{{ range .Alerts.Resolved }}
---
*Alert:* `{{ .Labels.alertname }}`
{{ if .Labels.instance }}*Instance:* `{{ .Labels.instance }}`{{ end }}
*Status:* ✅ *RESOLVED* at {{ .EndsAt.Format "2006-01-02 15:04:05 MST" }} (was active since {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }})
{{ if .Annotations.summary }}*Summary:* {{ .Annotations.summary }}{{ end }}
{{ if .Annotations.description }}*Description:* {{ .Annotations.description }}{{ end }}
*Labels:*
{{ range .Labels.SortedPairs }} • `{{ .Name }}` = `{{ .Value }}`
{{ end }}
{{ if gt (len .Annotations) 0 }}*Annotations:*
{{ range .Annotations.SortedPairs }} • `{{ .Name }}` = `{{ .Value }}`
{{ end }}{{ end }}
{{ if .DashboardURL }}📊 [View Dashboard]({{ .DashboardURL }})
{{ end }}{{ if .PanelURL }}📈 [View Panel]({{ .PanelURL }})
{{ end }}{{ if .GeneratorURL }}🔗 [Alert Source]({{ .GeneratorURL }}) {{ end }} {{ end }} {{ end }}
{{ if or (gt (len .Alerts.Firing) 0) (gt (len .Alerts.Resolved) 0) }}
🔔 *Grafana Alertmanager:* [View All Alerts]({{ template "__alertmanagerURL" . }}) 🔔
{{ end }}
{{ end }}
infrastructure/grafana/provisioning/dashboards/AdNova/statistics.json
+634 -638
View File
File diff suppressed because it is too large Load Diff
infrastructure/grafana/provisioning/dashboards/Backend/overview.json
+1021
View File
File diff suppressed because it is too large Load Diff
infrastructure/grafana/provisioning/dashboards/Backend/requests-by-view.json
+910
View File
@@ -0,0 +1,910 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "A dashboard that monitors Django which focuses on breaking down requests by view.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 12,
"links": [
{
"tags": ["backend"],
"targetBlank": true,
"title": "Backend Dashboards",
"type": "dashboards"
}
],
"liveNow": true,
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 1,
"panels": [],
"title": "Summary",
"type": "row"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red"
},
{
"color": "yellow",
"value": 0.95
},
{
"color": "green",
"value": 0.99
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 0,
"y": 1
},
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum(\n rate(\n django_http_responses_total_by_status_view_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\",\n status!~\"[4-5].*\"\n }[1w]\n )\n) /\nsum(\n rate(\n django_http_responses_total_by_status_view_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\"\n }[1w]\n )\n)\n",
"refId": "A"
}
],
"title": "Success Rate (non 4xx-5xx responses) [1w]",
"type": "stat"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "yellow",
"value": 10
},
{
"color": "red",
"value": 100
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 6,
"y": 1
},
"id": 3,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum by (view) (\n increase(\n django_http_exceptions_total_by_view_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n }[1w]\n ) > 0\n)\n",
"refId": "A"
}
],
"title": "HTTP Exceptions [1w]",
"type": "stat"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "yellow",
"value": 1000
},
{
"color": "red",
"value": 2000
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 1
},
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.50,\n sum (\n rate (\n django_http_requests_latency_seconds_by_view_method_bucket {\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\"\n }[$__range]\n )\n ) by (job, le)\n)\n",
"refId": "A"
}
],
"title": "Average Request Latency (P50) [1w]",
"type": "stat"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "yellow",
"value": 2500
},
{
"color": "red",
"value": 5000
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 1
},
"id": 5,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.95,\n sum (\n rate (\n django_http_requests_latency_seconds_by_view_method_bucket {\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\"\n }[$__range]\n )\n ) by (job, le)\n)\n",
"refId": "A"
}
],
"title": "Average Request Latency (P95) [1w]",
"type": "stat"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 5
},
"id": 6,
"panels": [],
"title": "Request & Responses",
"type": "row"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 100,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 6
},
"id": 7,
"options": {
"legend": {
"calcs": ["lastNotNull", "mean", "max"],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"sortBy": "Mean",
"sortDesc": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "round(\n sum(\n rate(\n django_http_requests_total_by_view_transport_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\"\n }[$__rate_interval]\n ) > 0\n ) by (job), 0.001\n)\n",
"legendFormat": "reqps",
"refId": "A"
}
],
"title": "Requests",
"type": "timeseries"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 100,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "percent"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "2xx"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "green",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "3xx"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "blue",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "4xx"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "yellow",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "5xx"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "red",
"mode": "fixed"
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 6
},
"id": 8,
"options": {
"legend": {
"calcs": ["lastNotNull", "mean", "max"],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"sortBy": "Mean",
"sortDesc": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "round(\n sum(\n rate(\n django_http_responses_total_by_status_view_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n status=~\"2.*\",\n }[$__rate_interval]\n ) > 0\n ) by (job), 0.001\n)\n",
"legendFormat": "2xx",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "round(\n sum(\n rate(\n django_http_responses_total_by_status_view_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n status=~\"3.*\",\n }[$__rate_interval]\n ) > 0\n ) by (job), 0.001\n)\n",
"legendFormat": "3xx",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "round(\n sum(\n rate(\n django_http_responses_total_by_status_view_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n status=~\"4.*\",\n }[$__rate_interval]\n ) > 0\n ) by (job), 0.001\n)\n",
"legendFormat": "4xx",
"refId": "C"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "round(\n sum(\n rate(\n django_http_responses_total_by_status_view_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n status=~\"5.*\",\n }[$__rate_interval]\n ) > 0\n ) by (job), 0.001\n)\n",
"legendFormat": "5xx",
"refId": "D"
}
],
"title": "Responses",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 14
},
"id": 9,
"panels": [],
"title": "Latency & Status Codes",
"type": "row"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 100,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "value"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 15
},
"id": 10,
"options": {
"legend": {
"calcs": ["lastNotNull", "mean", "max"],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"sortBy": "Mean",
"sortDesc": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "round(\n sum(\n rate(\n django_http_responses_total_by_status_view_method_total{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\",\n }[$__rate_interval]\n ) > 0\n ) by (namespace, job, view, status, method), 0.001\n)\n",
"legendFormat": "{{ view }} / {{ status }} / {{ method }}",
"refId": "A"
}
],
"title": "Responses Status Codes",
"type": "timeseries"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 15
},
"id": 11,
"options": {
"legend": {
"calcs": ["lastNotNull", "mean", "max"],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"sortBy": "Mean",
"sortDesc": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.50,\n sum(\n irate(\n django_http_requests_latency_seconds_by_view_method_bucket{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\"\n }[$__rate_interval]\n ) > 0\n ) by (view, le)\n)\n",
"legendFormat": "50 - {{ view }}",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.95,\n sum(\n irate(\n django_http_requests_latency_seconds_by_view_method_bucket{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\"\n }[$__rate_interval]\n ) > 0\n ) by (view, le)\n)\n",
"legendFormat": "95 - {{ view }}",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.99,\n sum(\n irate(\n django_http_requests_latency_seconds_by_view_method_bucket{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\"\n }[$__rate_interval]\n ) > 0\n ) by (view, le)\n)\n",
"legendFormat": "99 - {{ view }}",
"refId": "C"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.999,\n sum(\n irate(\n django_http_requests_latency_seconds_by_view_method_bucket{\n namespace=~\"$namespace\",\n job=~\"$job\",\n view=\"$view\",\n method=~\"$method\"\n }[$__rate_interval]\n ) > 0\n ) by (view, le)\n)\n",
"legendFormat": "99.9 - {{ view }}",
"refId": "D"
}
],
"title": "Request Latency",
"type": "timeseries"
}
],
"preload": false,
"refresh": "5s",
"schemaVersion": 41,
"tags": ["backend"],
"templating": {
"list": [
{
"label": "Data source",
"name": "datasource",
"query": "prometheus",
"refresh": 1,
"type": "datasource"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": false,
"label": "Namespace",
"name": "namespace",
"query": "label_values(django_http_responses_total_by_status_view_method_total{}, namespace)",
"refresh": 2,
"sort": 1,
"type": "query"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": false,
"label": "Job",
"name": "job",
"query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\"}, job)",
"refresh": 2,
"sort": 1,
"type": "query"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": false,
"label": "View",
"name": "view",
"query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view!~\"<unnamed view>|health_check:health_check_home|prometheus-django-metrics\"}, view)",
"refresh": 2,
"sort": 1,
"type": "query"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": true,
"label": "Method",
"multi": true,
"name": "method",
"query": "label_values(django_http_responses_total_by_status_view_method_total{namespace=~\"$namespace\", job=~\"$job\", view=~\"$view\"}, method)",
"refresh": 2,
"sort": 1,
"type": "query"
}
]
},
"time": {
"from": "now-30m",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Requests / By View",
"uid": "backend-requests-by-view",
"version": 2,
"weekStart": "monday"
}
infrastructure/grafana/provisioning/dashboards/Backend/requests-overview.json
+1254
View File
File diff suppressed because it is too large Load Diff
infrastructure/grafana/provisioning/dashboards/Caddy/overview.json
+2159
View File
File diff suppressed because it is too large Load Diff
infrastructure/grafana/provisioning/dashboards/Celery/by-task.json
+768
View File
@@ -0,0 +1,768 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "A dashboard that monitors Celery.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 2,
"links": [
{
"tags": ["celery"],
"targetBlank": true,
"title": "Celery Dashboards",
"type": "dashboards"
}
],
"liveNow": true,
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 1,
"panels": [],
"title": "Tasks",
"type": "row"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"noValue": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Success Rate"
},
"properties": [
{
"id": "unit",
"value": "percentunit"
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 16,
"x": 0,
"y": 1
},
"id": 2,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"enablePagination": true,
"fields": "",
"reducer": ["sum"],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "Succeeded"
}
]
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_succeeded_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name)\n/(sum (\n round(\n increase(\n celery_task_succeeded_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name)\n+sum (\n round(\n increase(\n celery_task_failed_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name)\n) > -1\n",
"format": "table",
"instant": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_succeeded_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name) > 0\n",
"format": "table",
"instant": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_failed_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name) > 0\n",
"format": "table",
"instant": true,
"refId": "C"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_sent_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name) > 0\n",
"format": "table",
"instant": true,
"refId": "D"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_received_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name) > 0\n",
"format": "table",
"instant": true,
"refId": "E"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_rejected_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name) > 0\n",
"format": "table",
"instant": true,
"refId": "F"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_retried_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name) > 0\n",
"format": "table",
"instant": true,
"refId": "G"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_revoked_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n )\n) by (name) > 0\n",
"format": "table",
"instant": true,
"refId": "H"
}
],
"title": "Task Stats",
"transformations": [
{
"id": "merge"
},
{
"id": "organize",
"options": {
"excludeByName": {
"Time": true
},
"indexByName": {
"Value #A": 1,
"Value #B": 2,
"Value #C": 3,
"Value #D": 4,
"Value #E": 5,
"Value #F": 6,
"Value #G": 7,
"Value #H": 8,
"name": 0
},
"renameByName": {
"Value #A": "Success Rate",
"Value #B": "Succeeded",
"Value #C": "Failed",
"Value #D": "Sent",
"Value #E": "Received",
"Value #F": "Rejected",
"Value #G": "Retried",
"Value #H": "Revoked",
"name": "Name"
}
}
}
],
"type": "table"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 1
},
"id": 3,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"enablePagination": true,
"fields": "",
"reducer": ["sum"],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "Value"
}
]
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "round(\n sum (\n increase(\n celery_task_failed_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__range]\n )\n ) by (name, exception) > 0\n)\n",
"format": "table",
"instant": true,
"refId": "A"
}
],
"title": "Task Exceptions",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {
"Time": true,
"job": true
},
"indexByName": {
"Value": 2,
"exception": 1,
"name": 0
},
"renameByName": {
"exception": "Exception",
"name": "Task"
}
}
}
],
"type": "table"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 9
},
"id": 4,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"sortBy": "Mean",
"sortDesc": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_succeeded_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name) > 0\n",
"legendFormat": "Succeeded - {{ name }}",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_failed_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name) > 0\n",
"legendFormat": "Failed - {{ name }}",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_sent_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name) > 0\n",
"legendFormat": "Sent - {{ name }}",
"refId": "C"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_received_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name) > 0\n",
"legendFormat": "Received - {{ name }}",
"refId": "D"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_retried_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name) > 0\n",
"legendFormat": "Retried - {{ name }}",
"refId": "E"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_revoked_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name) > 0\n",
"legendFormat": "Revoked - {{ name }}",
"refId": "F"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_rejected_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name) > 0\n",
"legendFormat": "Rejected - {{ name }}",
"refId": "G"
}
],
"title": "Tasks Completed",
"type": "timeseries"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 17
},
"id": 5,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"sortBy": "Mean",
"sortDesc": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.0.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "sum (\n round(\n increase(\n celery_task_failed_total{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n )\n )\n) by (name, exception) > 0\n",
"legendFormat": "{{ name }}/{{ exception }}",
"refId": "A"
}
],
"title": "Task Exceptions",
"type": "timeseries"
},
{
"datasource": {
"type": "datasource",
"uid": "-- Mixed --"
},
"fieldConfig": {
"defaults": {
"custom": {
"spanNulls": false
},
"unit": "s"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "P50"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "green",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "P95"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "yellow",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "P99"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "red",
"mode": "fixed"
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 25
},
"id": 6,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "right",
"showLegend": true,
"sortBy": "Mean",
"sortDesc": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "v11.1.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.50,\n sum(\n irate(\n celery_task_runtime_bucket{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n ) > 0\n ) by (name, job, le)\n)\n",
"legendFormat": "P50 - {{ name }}",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.95,\n sum(\n irate(\n celery_task_runtime_bucket{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n ) > 0\n ) by (name, job, le)\n)\n",
"legendFormat": "P95 - {{ name }}",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"expr": "histogram_quantile(0.99,\n sum(\n irate(\n celery_task_runtime_bucket{\n job=\"$job\",\n name=~\"$task\",\n queue_name=~\"$queue_name\"\n }[$__rate_interval]\n ) > 0\n ) by (name, job, le)\n)\n",
"legendFormat": "P99 - {{ name }}",
"refId": "C"
}
],
"title": "Tasks Runtime",
"type": "timeseries"
}
],
"preload": false,
"refresh": "10s",
"schemaVersion": 41,
"tags": ["celery"],
"templating": {
"list": [
{
"label": "Data source",
"name": "datasource",
"query": "prometheus",
"refresh": 1,
"type": "datasource"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": false,
"label": "Namespace",
"name": "namespace",
"query": "label_values(celery_worker_up{}, namespace)",
"refresh": 2,
"sort": 1,
"type": "query"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": false,
"label": "Job",
"name": "job",
"query": "label_values(celery_worker_up{namespace=\"$namespace\"}, job)",
"refresh": 2,
"sort": 1,
"type": "query"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": false,
"label": "Queue Name",
"name": "queue_name",
"query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", name!~\"None\"}, queue_name)",
"refresh": 2,
"sort": 1,
"type": "query"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"includeAll": false,
"label": "Task",
"multi": true,
"name": "task",
"query": "label_values(celery_task_received_total{namespace=\"$namespace\", job=\"$job\", queue_name=~\"$queue_name\", name!~\"None\"}, name)",
"refresh": 2,
"sort": 1,
"type": "query"
}
]
},
"time": {
"from": "now-24h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "By Task",
"uid": "celery-tasks-by-task",
"version": 1,
"weekStart": "monday"
}
infrastructure/grafana/provisioning/dashboards/Celery/overview.json
+1224
View File
File diff suppressed because it is too large Load Diff
infrastructure/grafana/provisioning/dashboards/Postgres/overview.json
+1416
View File
File diff suppressed because it is too large Load Diff
infrastructure/grafana/provisioning/dashboards/Redis/overview.json
+1551
View File
File diff suppressed because it is too large Load Diff
infrastructure/grafana/provisioning/dashboards/providers.yaml
+3 -1
View File
@@ -1,11 +1,13 @@
apiVersion: 1
providers:
- name: "celery"
- name: "default"
orgId: 1
folder: ""
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: false
options:
path: /etc/grafana/provisioning/dashboards
foldersFromFilesStructure: true
infrastructure/grafana/provisioning/datasources/datasources.yaml
+20 -1
View File
@@ -1,10 +1,29 @@
apiVersion: 1
datasources:
- name: Infinity
- name: Backend | Infinity
type: yesoreyeram-infinity-datasource
access: proxy
orgId: 1
uid: infinity
url: http://backend:8080
basicAuth: false
basicAuthUser: ""
withCredentials: false
isDefault: false
editable: false
- name: Prometheus
type: prometheus
access: proxy
orgId: 1
uid: prometheus
url: http://prometheus:9090
basicAuth: false
basicAuthUser: ""
withCredentials: false
isDefault: true
editable: false
jsonData:
httpMethod: POST
queryTimeout: 10s
timeInterval": 10s
infrastructure/grafana/provisioning/plugins/plugings.yaml → infrastructure/grafana/provisioning/plugins/plugins.yaml
View File
infrastructure/loadtest/.env.template
+1
View File
@@ -0,0 +1 @@
BACKEND_ADDRESS=http://backend:8080
infrastructure/pgadmin/.env.template
+3 -1
View File
@@ -1,2 +1,4 @@
PGADMIN_DEFAULT_EMAIL=admin@mail.com
PGADMIN_DEFAULT_PASSWORD_FILE=/run/secrets/pgadmin_password
PGADMIN_DEFAULT_PASSWORD=password
PGADMIN_DISABLE_POSTFIX=True
PGADMIN_REPLACE_SERVERS_ON_STARTUP=True
infrastructure/postgres/.env.template
+1 -1
View File
@@ -1,3 +1,3 @@
POSTGRES_DB=postgres
POSTGRES_USER=postgres
POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
POSTGRES_PASSWORD=postgres
infrastructure/prometheus/prometheus.yaml
+13 -6
View File
@@ -1,7 +1,7 @@
# Global config
global:
scrape_interval: 10s
scrape_timeout: 10s
scrape_interval: 5s
scrape_timeout: 5s
evaluation_interval: 10m
external_labels:
environment: local
@@ -10,24 +10,31 @@ global:
scrape_configs:
# Prometheus
- job_name: prometheus
scrape_interval: 5s
static_configs:
- targets: ["localhost:9090"]
# Postgres
- job_name: postgres
scrape_interval: 10s
static_configs:
- targets: ["postgres-exporter:9187"]
# Redis
- job_name: redis
scrape_interval: 10s
static_configs:
- targets: ["redis-exporter:9121"]
# Celery
- job_name: celery
scrape_interval: 30s
scrape_interval: 15s
static_configs:
- targets: ["celery-exporter:9808"]
# Backend
- job_name: backend
static_configs:
- targets: ["backend:8080"]
# Caddy
- job_name: caddy
static_configs:
- targets: ["proxy:2019"]
infrastructure/telegram_bot/.env.template
+1 -1
View File
@@ -1,4 +1,4 @@
AIOGRAM_BOT_TOKEN=
AIOGRAM_BACKEND_URL=http://backend:8080
REDIS_URI=redis://redis:6379
MINIO_ENDPOINT=minio:9000
MINIO_ENDPOINT=http://minio:9000
services/backend/.dockerignore
+3
View File
@@ -183,3 +183,6 @@ Dockerfile.staticfiles
# Collected static files
static
# Profile files
*.prof
services/backend/.gitignore
+3
View File
@@ -171,3 +171,6 @@ cython_debug/
# Collected static files
static
# Profile files
*.prof
services/backend/Dockerfile
+6 -5
View File
@@ -1,7 +1,7 @@
# Stage 1: Install dependencies
FROM docker.io/python:3.11-alpine3.20 AS builder
FROM docker.io/python:3.13-alpine3.22 AS builder
COPY --from=ghcr.io/astral-sh/uv:0.4.30 /uv /uvx /bin/
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
WORKDIR /app
@@ -17,7 +17,7 @@ RUN uv sync --no-dev --no-install-project --no-cache
# Stage 2: Start the application
FROM docker.io/python:3.11-alpine3.20
FROM docker.io/python:3.13-alpine3.22
WORKDIR /app
@@ -32,11 +32,12 @@ USER app
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PYTHONOPTIMIZE=2 \
PATH="/opt/venv/bin:$PATH"
PATH="/opt/venv/bin:$PATH" \
DJANGO_SETTINGS_MODULE=config.settings
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --start-interval=2s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://127.0.0.1:8080/health?format=json || exit 1
CMD gunicorn config.wsgi --workers=8 -b 0.0.0.0:8080 --access-logfile - --error-logfile -
CMD [ "opentelemetry-instrument", "--service_name", "backend-django", "--traces_exporter", "zipkin_json", "gunicorn", "config.wsgi", "--workers=2", "-b", "0.0.0.0:8080", "--access-logfile", "-", "--error-logfile", "-" ]
services/backend/Dockerfile.staticfiles
+6 -5
View File
@@ -1,7 +1,7 @@
# Stage 1: Install dependencies and compile staticfiles
FROM docker.io/python:3.11-alpine3.20 AS builder
FROM docker.io/python:3.13-alpine3.22 AS builder
COPY --from=ghcr.io/astral-sh/uv:0.4.30 /uv /uvx /bin/
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
WORKDIR /app
@@ -17,11 +17,12 @@ RUN uv sync --no-dev --no-install-project --no-cache
COPY . .
RUN uv run python manage.py collectstatic --noinput
RUN uv run --no-dev python manage.py collectstatic --noinput
# Stage 2: Start nginx and serve staticfiles
FROM docker.io/nginx:latest
FROM docker.io/nginx:1.29-alpine-slim
COPY --from=builder /app/static /usr/share/nginx/html
CMD ["nginx", "-g", "daemon off;"]
CMD [ "nginx", "-g", "daemon off;" ]
services/backend/README.md
+3 -3
View File
@@ -4,9 +4,9 @@
Ensure you have the following installed on your system:
- [Python](https://www.python.org/) (>=3.10,<3.12)
- [uv](https://docs.astral.sh/uv/)
- [Docker](https://www.docker.com/) (for containerized setup)
- [Python](https://www.python.org/) (>=3.10,<3.14)
- [uv](https://docs.astral.sh/uv/) (latest version recommended)
- [Docker](https://www.docker.com/) (for containerized setup, latest version recommended)
## Basic setup
services/backend/api/v1/ads/views.py
+2
View File
@@ -4,6 +4,7 @@ from uuid import UUID
from django.http import Http404, HttpRequest
from django.shortcuts import get_object_or_404
from ninja import Router
from silk.profiling.profiler import silk_profile
from api.v1 import schemas as global_schemas
from api.v1.ads import schemas
@@ -21,6 +22,7 @@ router = Router(tags=["ads"])
status.NOT_FOUND: global_schemas.NotFoundError,
},
)
@silk_profile("Get Advertisment")
def get_advertisment(
request: HttpRequest, client_id: UUID
) -> tuple[status, Campaign]:
services/backend/api/v1/generate/views.py
+5 -1
View File
@@ -49,8 +49,12 @@ def get_generate_ad_text_result(
if task_result.status == celery.states.PENDING:
raise Http404
result = task_result.result
if task_result.status != celery.states.SUCCESS:
result = None
return status.OK, schemas.Promise(
task_id=task_result.task_id,
status=task_result.status,
result=task_result.result,
result=result,
)
services/backend/api/v1/time/__init__.py
-1
View File
@@ -1 +0,0 @@
# noqa: A005
services/backend/apps/advertiser/models.py
+4 -4
View File
@@ -41,10 +41,10 @@ class Advertiser(BaseModel):
(
Decimal(str(total_clicks))
/ Decimal(str(total_impressions))
* Decimal("100")
* Decimal(100)
)
if total_impressions > 0
else Decimal("0")
else Decimal(0)
)
return {
@@ -115,9 +115,9 @@ class Advertiser(BaseModel):
conversion = (
Decimal(str(metrics["clicks_count"]))
/ Decimal(str(metrics["impressions_count"]))
* Decimal("100")
* Decimal(100)
if metrics["impressions_count"] > 0
else Decimal("0")
else Decimal(0)
)
daily_stats.append(
services/backend/apps/campaign/models.py
+2 -2
View File
@@ -256,10 +256,10 @@ class Campaign(BaseModel):
(
Decimal(str(clicks_count))
/ Decimal(str(impressions_count))
* Decimal("100")
* Decimal(100)
)
if impressions_count > 0
else Decimal("0")
else Decimal(0)
)
spent_impressions = Decimal(str(impressions.get("spent", 0) or 0))
spent_clicks = Decimal(str(clicks.get("spent", 0) or 0))
services/backend/config/settings.py
+81 -1
View File
@@ -4,6 +4,7 @@ import contextlib
import logging
from collections.abc import Callable
from pathlib import Path
from typing import TYPE_CHECKING
import django_stubs_ext
import environ
@@ -12,6 +13,10 @@ from health_check.plugins import plugin_dir
from integrations.yandexai.healthcheck import YandexAIHealthCheck
if TYPE_CHECKING:
from django.contrib.auth.models import User
BASE_DIR = Path(__file__).resolve().parent.parent
env = environ.Env()
@@ -53,7 +58,7 @@ REDIS_URI = env("REDIS_URI", default="redis://localhost:6379")
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.redis.RedisCache",
"BACKEND": "django_prometheus.cache.backends.redis.RedisCache",
"LOCATION": REDIS_URI,
"TIMEOUT": None,
"KEY_PREFIX": "backend",
@@ -80,6 +85,9 @@ CELERY_TASK_TRACK_STARTED = True
# Database
DB_URI = env.db_url("DJANGO_DB_URI", default="sqlite:///db.sqlite3")
DB_URI["ENGINE"] = DB_URI["ENGINE"].replace(
"django.db.backends", "django_prometheus.db.backends"
)
DATABASES = {"default": {**DB_URI, "CONN_MAX_AGE": 50}}
@@ -284,12 +292,15 @@ INTERNAL_IPS = env(
)
MIDDLEWARE = [
"silk.middleware.SilkyMiddleware",
"django_prometheus.middleware.PrometheusBeforeMiddleware",
"django_guid.middleware.guid_middleware",
"corsheaders.middleware.CorsMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"django_prometheus.middleware.PrometheusAfterMiddleware",
]
SIGNING_BACKEND = "django.core.signing.TimestampSigner"
@@ -442,8 +453,10 @@ INSTALLED_APPS = [
"corsheaders",
"django_extensions",
"django_guid",
"django_prometheus",
"ninja",
"minio_storage",
"silk",
# Internal apps
"apps.core",
"apps.advertiser",
@@ -516,6 +529,13 @@ SECRET_KEY = env("DJANGO_SECRET_KEY", default="very_insecure_key")
SECRET_KEY_FALLBACKS: list[str] = []
# Auth
LOGIN_REDIRECT_URL = "/admin/"
LOGIN_URL = "/admin/"
# Sessions
SESSION_CACHE_ALIAS = "default"
@@ -594,3 +614,63 @@ DEBUG_TOOLBAR_CONFIG = {"SHOW_COLLAPSED": True, "UPDATE_ON_FETCH": True}
if DEBUG and DEBUG_TOOLBAR_ENABLED:
INSTALLED_APPS.append("debug_toolbar")
MIDDLEWARE.append("debug_toolbar.middleware.DebugToolbarMiddleware")
# Prometheus
PROMETHEUS_LATENCY_BUCKETS = (
0.005,
0.01,
0.025,
0.05,
0.075,
0.1,
0.25,
0.5,
0.75,
1.0,
2.5,
5.0,
7.5,
10.0,
25.0,
50.0,
75.0,
float("inf"),
)
# django-silk
SILKY_PYTHON_PROFILER = True
SILKY_PYTHON_PROFILER_BINARY = True
SILKY_PYTHON_PROFILER_RESULT_PATH = "./profiles"
SILKY_PYTHON_PROFILER_EXTENDED_FILE_NAME = True
SILKY_AUTHENTICATION = True
SILKY_AUTHORISATION = True
def is_allowed_to_use_profiling(user: "User") -> bool:
return user.is_staff
SILKY_PERMISSIONS = is_allowed_to_use_profiling
SILKY_MAX_RECORDED_REQUESTS = 10**3
SILKY_MAX_RECORDED_REQUESTS_CHECK_PERCENT = 10
SILKY_MAX_REQUEST_BODY_SIZE = 128
SILKY_INTERCEPT_PERCENT = 25
SILKY_META = True
SILKY_DYNAMIC_PROFILING = [
{"module": "api.v1.ads.views", "function": "get_advertisment"}
]
services/backend/config/urls.py
+4
View File
@@ -16,6 +16,10 @@ urlpatterns = [
path("admin/", admin.site.urls),
# API urls
path("", include("api.urls")),
# Prometheus urls
path("", include("django_prometheus.urls")),
# Django-silk
path("silk/", include("silk.urls", namespace="silk")),
]
services/backend/integrations/yandexai/moderation.py
+1 -1
View File
@@ -1,4 +1,4 @@
# ruff: noqa: E501, W291
# ruff: noqa: E501
import logging
from django.conf import settings
services/backend/manage.py
+1
View File
@@ -1,4 +1,5 @@
#!/usr/bin/env python
# ruff: noqa: PLC0415
"""Django's command-line utility for administrative tasks."""
import os
services/backend/profiles/.gitkeep
View File
services/backend/pyproject.toml
+89 -69
View File
@@ -1,38 +1,58 @@
[project]
name = "adnova-backend"
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.10,<3.12"
dependencies = [
"celery>=5.4.0",
"colorlog>=6.9.0",
"django-cors-headers>=4.6.0",
"django-environ>=0.11.2",
"django-extensions>=3.2.3",
"django-guid>=3.5.0",
"django-health-check>=3.18.3",
"django-minio-storage>=0.5.7",
"django-ninja>=1.3.0",
"django-stubs-ext>=5.1.3",
"gunicorn>=23.0.0",
"httpx>=0.28.1",
"pillow>=11.1.0",
"psycopg2-binary>=2.9.10",
"pydantic>=2.10.5",
"pyjwt>=2.10.1",
"python-json-logger>=3.2.1",
"pytz>=2024.2",
"redis>=5.2.1",
"yandex-cloud-ml-sdk>=0.3.1",
"celery>=5.5.0,<6.0.0",
"colorlog>=6.9.0,<7.0.0",
"django-cors-headers>=4.7.0,<5.0.0",
"django-environ>=0.12.0,<1.0.0",
"django-extensions>=4.1.0,<5.0.0",
"django-guid>=3.5.1,<4.0.0",
"django-health-check>=3.18.3,<4.0.0",
"django-minio-storage>=0.5.7,<0.6.0",
"django-ninja>=1.3.0,<2.0.0",
"django-prometheus>=2.4.1,<3.0.0",
"django-redis>=6.0.0,<7.0.0",
"django-silk[formatting]>=5.4.0,<6.0.0",
"django-stubs-ext>=5.1.3,<6.0.0",
"gunicorn>=23.0.0,<24.0.0",
"httpx>=0.28.1,<0.29.0",
"opentelemetry-api>=1.35.0",
"opentelemetry-distro>=0.56b0",
"opentelemetry-exporter-otlp>=1.35.0",
"opentelemetry-exporter-zipkin-proto-http>=1.11.1",
"opentelemetry-instrumentation-asyncio>=0.56b0",
"opentelemetry-instrumentation-celery>=0.56b0",
"opentelemetry-instrumentation-dbapi>=0.56b0",
"opentelemetry-instrumentation-django>=0.56b0",
"opentelemetry-instrumentation-httpx>=0.56b0",
"opentelemetry-instrumentation-psycopg2>=0.56b0",
"opentelemetry-instrumentation-requests>=0.56b0",
"opentelemetry-instrumentation-sqlite3>=0.56b0",
"opentelemetry-instrumentation-threading>=0.56b0",
"opentelemetry-instrumentation-urllib>=0.56b0",
"opentelemetry-instrumentation-urllib3>=0.56b0",
"opentelemetry-instrumentation-wsgi>=0.56b0",
"opentelemetry-sdk>=1.35.0",
"pillow>=11.1.0,<12.0.0",
"psycopg2-binary>=2.9.10,<3.0.0",
"pydantic>=2.10.5,<3.0.0",
"pyjwt>=2.10.1,<3.0.0",
"python-json-logger>=3.2.1,<4.0.0",
"pytz>=2024.2,<2025.0",
"redis>=6.2.0,<7.0.0",
"yandex-cloud-ml-sdk>=0.3.1,<0.4.0",
]
name = "adnova-backend"
readme = "README.md"
requires-python = ">=3.10,<3.14"
version = "0.1.0"
[dependency-groups]
dev = [
"coverage>=7.6.12",
"django-debug-toolbar>=4.4.6",
"django-stubs[compatible-mypy]>=5.1.3",
"mypy>=1.15.0",
"ruff>=0.9.3",
"coverage",
"django-debug-toolbar>=5.2,<5.3",
"django-stubs[compatible-mypy]",
"mypy",
"ruff",
]
[tool.ruff]
@@ -66,7 +86,7 @@ extend-include = []
fix = false
fix-only = false
force-exclude = true
include = ["*.py", "*.pyi", "*.ipynb", "**/pyproject.toml"]
include = ["**/pyproject.toml", "*.ipynb", "*.py", "*.pyi"]
indent-width = 4
line-length = 79
namespace-packages = []
@@ -81,20 +101,20 @@ unsafe-fixes = false
[tool.ruff.analyze]
detect-string-imports = true
direction = "Dependencies"
exclude = []
include-dependencies = {}
preview = false
direction = "Dependencies"
exclude = []
include-dependencies = {}
preview = false
[tool.ruff.format]
docstring-code-format = true
docstring-code-format = true
docstring-code-line-length = 79
exclude = []
indent-style = "space"
line-ending = "lf"
preview = false
quote-style = "double"
skip-magic-trailing-comma = false
exclude = []
indent-style = "space"
line-ending = "lf"
preview = false
quote-style = "double"
skip-magic-trailing-comma = false
[tool.ruff.lint]
allowed-confusables = ["ℹ"]
@@ -109,26 +129,26 @@ extend-unsafe-fixes = []
external = []
fixable = ["ALL"]
ignore = [
"ARG",
"D",
"ANN401",
"COM812",
"DJ001",
"DJ007",
"FBT001",
"FBT002",
"N813",
"PLR2004",
"PT009",
"PT027",
"RUF001",
"S311",
"ANN401",
"ARG",
"COM812",
"D",
"DJ001",
"DJ007",
"FBT001",
"FBT002",
"N813",
"PLR2004",
"PT009",
"PT027",
"RUF001",
"S311",
]
logger-objects = []
per-file-ignores = {}
preview = false
select = ["ALL"]
task-tags = ["TODO", "FIXME", "HACK", "WORKOUT"]
task-tags = ["FIXME", "HACK", "TODO", "WORKOUT"]
typing-modules = []
unfixable = []
@@ -136,26 +156,26 @@ unfixable = []
max-args = 6
[tool.mypy]
plugins = ["mypy_django_plugin.main"]
ignore_missing_imports = true
strict = false
show_error_context = false
no_implicit_optional = false
no_implicit_optional = false
plugins = ["mypy_django_plugin.main"]
show_error_context = false
strict = false
[tool.django-stubs]
django_settings_module = "config.settings"
strict_settings = false
strict_settings = false
[tool.coverage.run]
omit = [
"manage.py",
"config/wsgi.py",
"config/asgi.py",
"config/urls.py",
"config/settings.py",
"config/handlers.py",
"config/errors.py",
"integrations/yandexai/*"
"config/asgi.py",
"config/errors.py",
"config/handlers.py",
"config/settings.py",
"config/urls.py",
"config/wsgi.py",
"integrations/yandexai/*",
"manage.py",
]
[tool.coverage.report]
services/backend/scripts/initdb
+1
View File
@@ -11,3 +11,4 @@ if [ "$DJANGO_CREATE_SUPERUSER" = "True" ]; then
fi
python manage.py init_cache
python manage.py silk_clear_request_log
services/loadtest/.dockerignore
+38
View File
@@ -0,0 +1,38 @@
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, built with `go test -c`
*.test
# Code coverage profiles and other test artifacts
*.out
coverage.*
*.coverprofile
profile.cov
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
go.work
go.work.sum
# env file
.env
# Editor/IDE
# .idea/
# .vscode/
# App binary
main
# Gitignore
.gitignore
services/loadtest/.env.template
+4
View File
@@ -0,0 +1,4 @@
# Change all vars before going to production and remove all comments (!)
# Below all environment variables and default values
BACKEND_ADDRESS=http://localhost:8080
services/loadtest/.gitignore
+35
View File
@@ -0,0 +1,35 @@
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, built with `go test -c`
*.test
# Code coverage profiles and other test artifacts
*.out
coverage.*
*.coverprofile
profile.cov
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
go.work
go.work.sum
# env file
.env
# Editor/IDE
# .idea/
# .vscode/
# App binary
main
services/loadtest/Dockerfile
+28
View File
@@ -0,0 +1,28 @@
ARG GOARCH=amd64
# Stage 1: Build go binary
FROM docker.io/golang:1.24-alpine AS build
WORKDIR /build
COPY . .
RUN go mod download
ARG GOARCH
RUN CGO_ENABLED=0 GOARCH=$GOARCH go build -o loadtest
# Stage 2: Run go binary
FROM docker.io/alpine:3.22
WORKDIR /app
COPY --from=build /build/loadtest .
COPY . .
EXPOSE 5001
CMD [ "./loadtest", "--port", "5001" ]
services/loadtest/README.md
+68
View File
@@ -0,0 +1,68 @@
# AdNova Loadtest
## Prerequisites
Ensure you have the following installed on your system:
- [Go](https://go.dev/) (1.24 recommended)
- [Docker](https://www.docker.com/) (for containerized setup, latest version recommended)
## Basic setup
### Installation
#### Clone the project
#### Go to the project directory
```bash
cd AdNova/services/loadtest
```
#### Customize environment
```bash
cp .env.template .env
```
And setup env vars according to your needs.
#### Install dependencies
```bash
go mod download
```
#### Running
```bash
go run main.go
```
## Containerized setup
### Clone the project
### Go to the project directory
```bash
cd AdNova/services/loadtest
```
### Build docker image
```bash
docker build -t adnova-loadtest .
```
### Customize environment
Customize environment with `docker run` command, for all environment vars and default values see [.env.template](./.env.template).
### Run docker image
```bash
docker run -p 5001:5001 --name adnova-loadtest adnova-loadtest
```
Loadtest will be available on [127.0.0.1:5001](http://127.0.0.1:5001).
services/loadtest/go.mod
+8
View File
@@ -0,0 +1,8 @@
module loadtest
go 1.24.5
require (
github.com/gorilla/mux v1.8.1
github.com/gorilla/websocket v1.5.3
)
services/loadtest/go.sum
+4
View File
@@ -0,0 +1,4 @@
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
services/loadtest/main.go
+788
View File
@@ -0,0 +1,788 @@
package main
import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"html/template"
"io"
"log"
"math/rand"
"net/http"
"os"
"strings"
"sync"
"time"
"github.com/gorilla/mux"
"github.com/gorilla/websocket"
)
// --- Mock Data Structures ---
type Client struct {
ClientID string `json:"client_id"`
Login string `json:"login"`
Age int `json:"age"`
Location string `json:"location"`
Gender string `json:"gender"`
}
type Advertiser struct {
AdvertiserID string `json:"advertiser_id"`
Name string `json:"name"`
}
type MLScore struct {
AdvertiserId string `json:"advertiser_id"`
ClientID string `json:"client_id"`
Score int64 `json:"score"`
}
type Campaign struct {
Targeting struct {
Gender string `json:"gender,omitempty"`
AgeFrom int `json:"age_from,omitempty"`
AgeTo int `json:"age_to,omitempty"`
Location string `json:"location,omitempty"`
} `json:"targeting"`
AdTitle string `json:"ad_title"`
AdText string `json:"ad_text"`
ImpressionsLimit int `json:"impressions_limit"`
ClicksLimit int `json:"clicks_limit"`
CostPerImpression float64 `json:"cost_per_impression"`
CostPerClick float64 `json:"cost_per_click"`
StartDate int64 `json:"start_date"`
EndDate int64 `json:"end_date"`
}
type CampaignFileEntry struct {
AdvertiserID string `json:"advertiser_id"`
CampaignData Campaign `json:"campaign_data"`
}
// --- Mock data filenames ---
const clientsMockDataFile = "./mocks/bulk_clients.json"
const advertisersMockDataFile = "./mocks/bulk_advertisers.json"
const campaignsMockDataFile = "./mocks/campaigns.json"
const mlscoresMockDataFile = "./mocks/ml_scores.json"
// --- In-Memory Data Store ---
var (
loadedClientIDs []string
dataMutex = &sync.RWMutex{}
backendAddress string
)
// --- Mock Data Loading and Posting Logic ---
func loadInitialMockIDs() {
dataMutex.Lock()
defer dataMutex.Unlock()
loadedClientIDs = nil
log.Println("Loading client IDs from local files...")
readFile := func(filename string) ([]byte, error) {
content, err := os.ReadFile(filename)
if err != nil {
return nil, fmt.Errorf("error reading %s: %w", filename, err)
}
return content, nil
}
clientsContent, err := readFile(clientsMockDataFile)
if err != nil {
log.Printf("Warning: Could not read %s for initial ID load: %v", clientsMockDataFile, err)
} else {
var tempClients []Client
if err := json.Unmarshal(clientsContent, &tempClients); err != nil {
log.Printf("Warning: Error unmarshaling %s for initial ID load: %v", clientsMockDataFile, err)
} else {
for _, c := range tempClients {
loadedClientIDs = append(loadedClientIDs, c.ClientID)
}
log.Printf("Loaded %d client IDs for stress testing.", len(loadedClientIDs))
}
}
}
func loadAndPostMocks() error {
log.Println("Attempting to load mock data from files and post to external backend:", backendAddress)
readFile := func(filename string) ([]byte, error) {
content, err := os.ReadFile(filename)
if err != nil {
return nil, fmt.Errorf("error reading %s: %w", filename, err)
}
return content, nil
}
postJSON := func(url string, data interface{}) error {
jsonData, err := json.Marshal(data)
if err != nil {
return fmt.Errorf("failed to marshal JSON for %s: %w", url, err)
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
if err != nil {
return fmt.Errorf("failed to create request for %s: %w", url, err)
}
req.Header.Set("Content-Type", "application/json")
client := &http.Client{
Timeout: 60 * time.Second,
}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("failed to POST to %s: %w", url, err)
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
body, _ := io.ReadAll(resp.Body)
return fmt.Errorf("POST to %s failed with status %d: %s", url, resp.StatusCode, string(body))
}
return nil
}
// 1. Load bulk_clients.json
clientsContent, err := readFile(clientsMockDataFile)
if err != nil {
return err
}
var tempClients []Client
if err := json.Unmarshal(clientsContent, &tempClients); err != nil {
return fmt.Errorf("error unmarshaling %s: %w", clientsMockDataFile, err)
}
if err := postJSON(fmt.Sprintf("%s/clients/bulk", backendAddress), tempClients); err != nil {
return fmt.Errorf("error posting bulk clients: %w", err)
}
log.Printf("Successfully posted %d clients to %s/clients/bulk", len(tempClients), backendAddress)
// 2. Load bulk_advertisers.json
advertisersContent, err := readFile(advertisersMockDataFile)
if err != nil {
return err
}
var tempAdvertisers []Advertiser
if err := json.Unmarshal(advertisersContent, &tempAdvertisers); err != nil {
return fmt.Errorf("error unmarshaling %s: %w", advertisersMockDataFile, err)
}
if err := postJSON(fmt.Sprintf("%s/advertisers/bulk", backendAddress), tempAdvertisers); err != nil {
return fmt.Errorf("error posting bulk advertisers: %w", err)
}
log.Printf("Successfully posted %d advertisers to %s/advertisers/bulk", len(tempAdvertisers), backendAddress)
// 3. Load campaigns.json
campaignsContent, err := readFile(campaignsMockDataFile)
if err != nil {
return err
}
var campaignDataList []CampaignFileEntry
if err := json.Unmarshal(campaignsContent, &campaignDataList); err != nil {
return fmt.Errorf("error unmarshaling %s: %w", campaignsMockDataFile, err)
}
for i, entry := range campaignDataList {
if err := postJSON(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, entry.AdvertiserID), entry.CampaignData); err != nil {
log.Printf("Warning: Failed to post campaign for advertiser %s (entry %d): %v", entry.AdvertiserID, i, err)
}
}
log.Printf("Attempted to post campaigns for %d advertisers from %s", len(campaignDataList), campaignsMockDataFile)
// 4. Load ml_scores.json
mlScoresContent, err := readFile(mlscoresMockDataFile)
if err != nil {
return err
}
var tempMLScores []MLScore
if err := json.Unmarshal(mlScoresContent, &tempMLScores); err != nil {
return fmt.Errorf("error unmarshaling %s: %w", mlscoresMockDataFile, err)
}
for i, score := range tempMLScores {
if err := postJSON(fmt.Sprintf("%s/ml-scores", backendAddress), score); err != nil {
log.Printf("Warning: Failed to post ML score %d (client %s): %v", i, score.ClientID, err)
}
}
log.Printf("Attempted to post %d ML scores one by one to %s/ml-scores", len(tempMLScores), backendAddress)
log.Println("Mock data loading and posting process complete.")
return nil
}
// --- WebSocket Hub ---
var upgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(r *http.Request) bool { return true },
}
type Hub struct {
clients map[*websocket.Conn]bool
broadcast chan []byte
register chan *websocket.Conn
unregister chan *websocket.Conn
mutex sync.Mutex
}
func newHub() *Hub {
return &Hub{
broadcast: make(chan []byte),
register: make(chan *websocket.Conn),
unregister: make(chan *websocket.Conn),
clients: make(map[*websocket.Conn]bool),
}
}
func (h *Hub) run() {
for {
select {
case client := <-h.register:
h.mutex.Lock()
h.clients[client] = true
h.mutex.Unlock()
case client := <-h.unregister:
h.mutex.Lock()
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
client.Close()
}
h.mutex.Unlock()
case message := <-h.broadcast:
h.mutex.Lock()
for client := range h.clients {
err := client.WriteMessage(websocket.TextMessage, message)
if err != nil {
log.Printf("error: %v", err)
client.Close()
delete(h.clients, client)
}
}
h.mutex.Unlock()
}
}
}
var hub = newHub()
// --- Load Generator ---
type TestConfig struct {
BackendAddress string `json:"backendAddress"`
MaxRPS int `json:"maxRps"`
LoadProfile string `json:"loadProfile"`
FromRPS int `json:"fromRPS"`
ToRPS int `json:"toRPS"`
StepRPS int `json:"stepRps"`
StepDuration int `json:"stepDuration"`
OnceCount int `json:"onceCount"`
}
type RequestResult struct {
StatusCode int
Latency time.Duration
Error bool
}
type TestStats struct {
RPS int `json:"rps"`
Latency float64 `json:"latency"`
ErrorRate float64 `json:"errorRate"`
TotalReqs int64 `json:"totalReqs"`
TotalErrors int64 `json:"totalErrors"`
IsRunning bool `json:"isRunning"`
}
type TestManager struct {
config TestConfig
isRunning bool
cancelFunc context.CancelFunc
mutex sync.Mutex
httpClient *http.Client
}
var testManager = TestManager{
httpClient: &http.Client{
Timeout: 10 * time.Second,
Transport: &http.Transport{
MaxIdleConns: 1000,
MaxIdleConnsPerHost: 1000,
IdleConnTimeout: 90 * time.Second,
},
},
}
func (tm *TestManager) startTest(config TestConfig) {
tm.mutex.Lock()
if tm.isRunning {
tm.mutex.Unlock()
log.Println("Test already running.")
return
}
tm.config = config
tm.isRunning = true
ctx, cancel := context.WithCancel(context.Background())
tm.cancelFunc = cancel
tm.mutex.Unlock()
go tm.runLoadGenerator(ctx)
}
func (tm *TestManager) stopTest() {
tm.mutex.Lock()
if tm.isRunning && tm.cancelFunc != nil {
tm.cancelFunc()
tm.isRunning = false
}
tm.mutex.Unlock()
}
func (tm *TestManager) runLoadGenerator(ctx context.Context) {
log.Printf("Starting test with config: %+v\n", tm.config)
results := make(chan RequestResult, 10000)
var wg sync.WaitGroup
var totalReqs, totalErrors int64
go func() {
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
var reqsInSecond int
var totalLatency time.Duration
var errorsInSecond int
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
avgLatency := 0.0
if reqsInSecond > 0 {
avgLatency = float64(totalLatency.Milliseconds()) / float64(reqsInSecond)
}
errorRate := 0.0
if reqsInSecond > 0 {
errorRate = float64(errorsInSecond) / float64(reqsInSecond) * 100
}
stats := TestStats{
RPS: reqsInSecond,
Latency: avgLatency,
ErrorRate: errorRate,
TotalReqs: totalReqs,
TotalErrors: totalErrors,
IsRunning: true,
}
jsonStats, _ := json.Marshal(stats)
hub.broadcast <- jsonStats
reqsInSecond = 0
totalLatency = 0
errorsInSecond = 0
case res, ok := <-results:
if !ok {
return
}
totalReqs++
reqsInSecond++
totalLatency += res.Latency
if res.Error {
totalErrors++
errorsInSecond++
}
}
}
}()
numWorkers := 1000
if tm.config.MaxRPS > 0 && tm.config.MaxRPS < numWorkers {
numWorkers = tm.config.MaxRPS
}
jobs := make(chan string, 10000)
for i := 0; i < numWorkers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for {
select {
case <-ctx.Done():
return
case url, ok := <-jobs:
if !ok {
return
}
start := time.Now()
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
resp, err := tm.httpClient.Do(req)
latency := time.Since(start)
if err != nil {
results <- RequestResult{StatusCode: 0, Latency: latency, Error: true}
continue
}
results <- RequestResult{StatusCode: resp.StatusCode, Latency: latency, Error: resp.StatusCode >= 500}
resp.Body.Close()
}
}
}()
}
go func() {
defer close(jobs)
dataMutex.RLock()
if len(loadedClientIDs) == 0 {
log.Println("No client IDs loaded for stress test.")
dataMutex.RUnlock()
return
}
clientIDsForTest := make([]string, len(loadedClientIDs))
copy(clientIDsForTest, loadedClientIDs)
dataMutex.RUnlock()
log.Printf("Stress testing with %d client IDs.", len(clientIDsForTest))
switch tm.config.LoadProfile {
case "const":
ticker := time.NewTicker(time.Second / time.Duration(tm.config.MaxRPS))
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
jobs <- url
}
}
case "line":
duration := 10 * time.Second
startTime := time.Now()
for {
select {
case <-ctx.Done():
return
default:
elapsed := time.Since(startTime)
if elapsed >= duration {
elapsed = duration
}
progress := float64(elapsed) / float64(duration)
currentRPS := float64(tm.config.FromRPS) + (float64(tm.config.ToRPS-tm.config.FromRPS) * progress)
if currentRPS <= 0 {
currentRPS = 1
}
sleepDuration := time.Second / time.Duration(currentRPS)
time.Sleep(sleepDuration)
clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
jobs <- url
if elapsed >= duration {
constSleepDuration := time.Second / time.Duration(tm.config.ToRPS)
for {
select {
case <-ctx.Done():
return
default:
time.Sleep(constSleepDuration)
clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
jobs <- url
}
}
}
}
}
case "step":
currentRPS := tm.config.FromRPS
for currentRPS <= tm.config.ToRPS {
log.Printf("Step load: %d RPS for %d seconds", currentRPS, tm.config.StepDuration)
stepEndTime := time.After(time.Duration(tm.config.StepDuration) * time.Second)
sleepDuration := time.Second / time.Duration(currentRPS)
stepLoop:
for {
select {
case <-ctx.Done():
return
case <-stepEndTime:
break stepLoop
default:
time.Sleep(sleepDuration)
clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
jobs <- url
}
}
currentRPS += tm.config.StepRPS
if tm.config.StepRPS == 0 && currentRPS != tm.config.ToRPS {
break
}
}
case "once":
for i := 0; i < tm.config.OnceCount; i++ {
select {
case <-ctx.Done():
return
default:
clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
jobs <- url
}
}
case "unlimited":
for {
select {
case <-ctx.Done():
return
default:
clientID := clientIDsForTest[rand.Intn(len(clientIDsForTest))]
url := fmt.Sprintf("%s/ads?client_id=%s", tm.config.BackendAddress, clientID)
jobs <- url
}
}
}
}()
wg.Wait()
close(results)
time.Sleep(1 * time.Second)
tm.mutex.Lock()
tm.isRunning = false
tm.mutex.Unlock()
finalStats := TestStats{IsRunning: false, TotalReqs: totalReqs, TotalErrors: totalErrors}
jsonStats, _ := json.Marshal(finalStats)
hub.broadcast <- jsonStats
log.Println("Test finished.")
}
// --- HTTP Handlers ---
func serveWs(w http.ResponseWriter, r *http.Request) {
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println(err)
return
}
hub.register <- conn
go func() {
defer func() {
hub.unregister <- conn
conn.Close()
}()
for {
if _, _, err := conn.NextReader(); err != nil {
break
}
}
}()
}
func handleLoadMocks(w http.ResponseWriter, r *http.Request) {
if err := loadAndPostMocks(); err != nil {
http.Error(w, fmt.Sprintf("Failed to load mocks: %v", err), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map[string]string{"status": "Mocks loaded successfully"})
}
func checkEndpoint(url string) bool {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
log.Printf("Error creating request for %s: %v", url, err)
return false
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
req = req.WithContext(ctx)
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Printf("Error making request to %s: %v", url, err)
return false
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Printf("Check for %s failed with status: %d", url, resp.StatusCode)
return false
}
if strings.Contains(url, "/campaigns") {
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
log.Printf("Error reading response body from %s: %v", url, err)
return false
}
var campaignsData []Campaign
if err := json.Unmarshal(bodyBytes, &campaignsData); err != nil {
log.Printf("Error unmarshaling campaign data from %s: %v", url, err)
return false
}
return len(campaignsData) > 0
}
return true
}
func handleCheckMocks(w http.ResponseWriter, r *http.Request) {
dataMutex.RLock()
defer dataMutex.RUnlock()
status := make(map[string]bool)
var tempClients []Client
clientsContent, err := os.ReadFile(clientsMockDataFile)
if err != nil {
log.Printf("Warning: Could not read %s for checks: %v", clientsMockDataFile, err)
} else {
json.Unmarshal(clientsContent, &tempClients)
}
var tempAdvertisers []Advertiser
advertisersContent, err := os.ReadFile(advertisersMockDataFile)
if err != nil {
log.Printf("Warning: Could not read %s for checks: %v", advertisersMockDataFile, err)
} else {
json.Unmarshal(advertisersContent, &tempAdvertisers)
}
clientsLoaded := false
if len(tempClients) >= 3 {
firstClient := tempClients[0].ClientID
medianClient := tempClients[len(tempClients)/2].ClientID
lastClient := tempClients[len(tempClients)-1].ClientID
clientsLoaded = checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, firstClient)) &&
checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, medianClient)) &&
checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, lastClient))
} else if len(tempClients) > 0 {
clientsLoaded = checkEndpoint(fmt.Sprintf("%s/clients/%s", backendAddress, tempClients[0].ClientID))
}
status["clients"] = clientsLoaded
advertisersLoaded := false
if len(tempAdvertisers) >= 3 {
firstAdvertiser := tempAdvertisers[0].AdvertiserID
medianAdvertiser := tempAdvertisers[len(tempAdvertisers)/2].AdvertiserID
lastAdvertiser := tempAdvertisers[len(tempAdvertisers)-1].AdvertiserID
advertisersLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, firstAdvertiser)) &&
checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, medianAdvertiser)) &&
checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, lastAdvertiser))
} else if len(tempAdvertisers) > 0 {
advertisersLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s", backendAddress, tempAdvertisers[0].AdvertiserID))
}
status["advertisers"] = advertisersLoaded
campaignsLoaded := false
if len(tempAdvertisers) >= 3 {
firstAdvertiser := tempAdvertisers[0].AdvertiserID
medianAdvertiser := tempAdvertisers[len(tempAdvertisers)/2].AdvertiserID
lastAdvertiser := tempAdvertisers[len(tempAdvertisers)-1].AdvertiserID
campaignsLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, firstAdvertiser)) &&
checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, medianAdvertiser)) &&
checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, lastAdvertiser))
} else if len(tempAdvertisers) > 0 {
campaignsLoaded = checkEndpoint(fmt.Sprintf("%s/advertisers/%s/campaigns", backendAddress, tempAdvertisers[0].AdvertiserID))
}
status["campaigns"] = campaignsLoaded
_, err = os.ReadFile(mlscoresMockDataFile)
status["ml_scores"] = err == nil
if !status["ml_scores"] {
log.Printf("Warning: ML Scores file not found or readable, assuming not loaded for check.")
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(status)
}
func handleStartTest(w http.ResponseWriter, r *http.Request) {
var config TestConfig
if err := json.NewDecoder(r.Body).Decode(&config); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
config.BackendAddress = backendAddress
go testManager.startTest(config)
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map[string]string{"status": "Test started"})
}
func handleStopTest(w http.ResponseWriter, r *http.Request) {
testManager.stopTest()
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map[string]string{"status": "Test stopped"})
}
func main() {
rand.Seed(time.Now().UnixNano())
port := flag.String("port", "5002", "Port to run the server on")
flag.Parse()
backendAddress = os.Getenv("BACKEND_ADDRESS")
if backendAddress == "" {
log.Println("BACKEND_ADDRESS environment variable not set. Defaulting to http://localhost:8080")
backendAddress = "http://localhost:8080"
}
loadInitialMockIDs()
go hub.run()
r := mux.NewRouter()
r.HandleFunc("/", serveUI)
r.HandleFunc("/ws", serveWs)
r.HandleFunc("/api/load-mocks", handleLoadMocks).Methods("POST")
r.HandleFunc("/api/check-mocks", handleCheckMocks).Methods("GET")
r.HandleFunc("/api/start-test", handleStartTest).Methods("POST")
r.HandleFunc("/api/stop-test", handleStopTest).Methods("POST")
addr := ":" + *port
log.Printf("Server starting on port %s. Open http://localhost%s\n", *port, addr)
if err := http.ListenAndServe(addr, r); err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
// --- Embedded HTML UI ---
func serveUI(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
t, err := template.ParseFiles("static/index.html")
if err != nil {
http.Error(w, "Could not load UI template", http.StatusInternalServerError)
return
}
err = t.Execute(w, nil)
if err != nil {
http.Error(w, "Failed to render UI", http.StatusInternalServerError)
}
}
services/loadtest/mocks/bulk_advertisers.json
+122
View File
@@ -0,0 +1,122 @@
[
{
"advertiser_id": "addf3407-9265-44b8-8e7e-6b015f63c47d",
"name": "Smart Inc"
},
{
"advertiser_id": "4d830921-f650-4f2f-869b-db48208dc952",
"name": "Advanced Technologies"
},
{
"advertiser_id": "752ddcd3-e944-4eaf-b1de-5e49977354a7",
"name": "Global Group"
},
{
"advertiser_id": "e04d35cd-e24b-4bfd-b5fa-6ac1c12ae8a0",
"name": "Global Media"
},
{
"advertiser_id": "8b7f547f-bab5-448d-9997-61cbbd6e3f1c",
"name": "Modern Systems"
},
{
"advertiser_id": "57dd000f-32fe-4a49-8dca-bfdc84fec7dd",
"name": "Advanced Technologies"
},
{
"advertiser_id": "22d28b06-32c5-434e-b05e-0654fa1cac09",
"name": "Global Corp"
},
{
"advertiser_id": "7ba54148-4885-48da-97ba-6ecf7b3a300b",
"name": "Tech Solutions"
},
{
"advertiser_id": "00ac5657-1a1a-48f1-815a-2ffdb72b82b8",
"name": "Elite Group"
},
{
"advertiser_id": "b9d5a8ce-ae23-4a93-946e-4bd85bddef5c",
"name": "Modern Marketing"
},
{
"advertiser_id": "3696b94d-6591-4e15-86f3-8c46ee535e27",
"name": "Advanced Systems"
},
{
"advertiser_id": "20024aab-2476-4d35-b96f-f47589f05474",
"name": "Premier Technologies"
},
{
"advertiser_id": "a68ef7d6-c50f-40d1-b70f-a32d7cdcd036",
"name": "Elite Partners"
},
{
"advertiser_id": "960e2896-3f34-43a3-8aa1-d984bb92b96e",
"name": "Future Inc"
},
{
"advertiser_id": "80a7f6a9-5939-4d74-9a7e-e455b9fa0fdd",
"name": "Modern Marketing"
},
{
"advertiser_id": "6bb59511-360d-49dc-9626-be2f6b9b3a0a",
"name": "Future Solutions"
},
{
"advertiser_id": "d7a27dc9-7d8d-4f25-bea4-2528f88238d1",
"name": "Advanced Solutions"
},
{
"advertiser_id": "e6cde283-ca4b-456c-b1fb-c2967918b5ef",
"name": "Digital Solutions"
},
{
"advertiser_id": "2a51ba38-bed3-43af-bbb4-99093b72e0d1",
"name": "Tech Group"
},
{
"advertiser_id": "95a9a488-8fee-4aa6-9c21-458f8f5523ca",
"name": "Elite Systems"
},
{
"advertiser_id": "2fb992b8-4441-47f5-9176-1da68526eafb",
"name": "Tech Inc"
},
{
"advertiser_id": "f0d160bd-18b4-4c88-917c-06b699abeeec",
"name": "Global Inc"
},
{
"advertiser_id": "d8147d8c-2bcd-4ac8-bab5-82e420509a7f",
"name": "Smart Inc"
},
{
"advertiser_id": "94209720-2c10-4a55-bc37-c28838d08012",
"name": "Advanced Corp"
},
{
"advertiser_id": "33fae4ce-d252-4b34-af2b-c11e770f65b5",
"name": "Smart Advertising"
},
{
"advertiser_id": "2c0261ba-bd3c-4b19-a53c-44efa10819e5",
"name": "Digital Marketing"
},
{
"advertiser_id": "0892fbb7-fbc6-45e9-b52f-494258a9c8f8",
"name": "Smart Group"
},
{
"advertiser_id": "055af498-567a-4a4e-b25f-60a57b9b591c",
"name": "Advanced Systems"
},
{
"advertiser_id": "0dfd7246-4cfa-4721-899b-e6dd9c483626",
"name": "Premier Advertising"
},
{
"advertiser_id": "32d75bc6-0779-4081-98b3-58273497c561",
"name": "Modern Partners"
}
]
services/loadtest/mocks/bulk_clients.json
+1465
View File
File diff suppressed because it is too large Load Diff
services/loadtest/mocks/campaigns.json
+572
View File
@@ -0,0 +1,572 @@
[
{
"advertiser_id": "addf3407-9265-44b8-8e7e-6b015f63c47d",
"campaign_data": {
"impressions_limit": 8,
"clicks_limit": 8,
"cost_per_impression": 1.5,
"cost_per_click": 2.5,
"ad_title": "Amazing Offer",
"ad_text": "Get great savings now!",
"start_date": 0,
"end_date": 15,
"targeting": {
"gender": "MALE",
"age_from": 25,
"age_to": 40,
"location": "Chicago"
}
}
},
{
"advertiser_id": "4d830921-f650-4f2f-869b-db48208dc952",
"campaign_data": {
"impressions_limit": 10,
"clicks_limit": 10,
"cost_per_impression": 1.2,
"cost_per_click": 2.8,
"ad_title": "Incredible Deal",
"ad_text": "Discover amazing quality!",
"start_date": 0,
"end_date": 20,
"targeting": {
"gender": "FEMALE",
"age_from": 30,
"age_to": 50,
"location": "New York"
}
}
},
{
"advertiser_id": "752ddcd3-e944-4eaf-b1de-5e49977354a7",
"campaign_data": {
"impressions_limit": 6,
"clicks_limit": 6,
"cost_per_impression": 1.8,
"cost_per_click": 2.2,
"ad_title": "Best Service",
"ad_text": "Buy unique advantages today!",
"start_date": 0,
"end_date": 22,
"targeting": {
"gender": "ALL",
"age_from": 18,
"age_to": 60,
"location": "Los Angeles"
}
}
},
{
"advertiser_id": "e04d35cd-e24b-4bfd-b5fa-6ac1c12ae8a0",
"campaign_data": {
"impressions_limit": 7,
"clicks_limit": 7,
"cost_per_impression": 1.1,
"cost_per_click": 2.9,
"ad_title": "Exclusive Opportunity",
"ad_text": "Try our new product now!",
"start_date": 0,
"end_date": 13,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "8b7f547f-bab5-448d-9997-61cbbd6e3f1c",
"campaign_data": {
"impressions_limit": 9,
"clicks_limit": 9,
"cost_per_impression": 1.6,
"cost_per_click": 2.4,
"ad_title": "Special Features",
"ad_text": "Experience our service today!",
"start_date": 0,
"end_date": 11,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "57dd000f-32fe-4a49-8dca-bfdc84fec7dd",
"campaign_data": {
"impressions_limit": 5,
"clicks_limit": 5,
"cost_per_impression": 1.3,
"cost_per_click": 2.1,
"ad_title": "Limited Time Offer",
"ad_text": "Get it before it's gone!",
"start_date": 0,
"end_date": 12,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "22d28b06-32c5-434e-b05e-0654fa1cac09",
"campaign_data": {
"impressions_limit": 6,
"clicks_limit": 6,
"cost_per_impression": 1.7,
"cost_per_click": 2.6,
"ad_title": "New Arrival",
"ad_text": "Check out our latest product!",
"start_date": 0,
"end_date": 14,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "7ba54148-4885-48da-97ba-6ecf7b3a300b",
"campaign_data": {
"impressions_limit": 8,
"clicks_limit": 8,
"cost_per_impression": 1.9,
"cost_per_click": 2.3,
"ad_title": "Discover More",
"ad_text": "Find out what we offer!",
"start_date": 0,
"end_date": 16,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "00ac5657-1a1a-48f1-815a-2ffdb72b82b8",
"campaign_data": {
"impressions_limit": 7,
"clicks_limit": 7,
"cost_per_impression": 1.4,
"cost_per_click": 2.8,
"ad_title": "Best Quality",
"ad_text": "Experience the best quality!",
"start_date": 0,
"end_date": 18,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "b9d5a8ce-ae23-4a93-946e-4bd85bddef5c",
"campaign_data": {
"impressions_limit": 9,
"clicks_limit": 9,
"cost_per_impression": 1.6,
"cost_per_click": 2.2,
"ad_title": "Limited Edition",
"ad_text": "Get it while it lasts!",
"start_date": 0,
"end_date": 15,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "3696b94d-6591-4e15-86f3-8c46ee535e27",
"campaign_data": {
"impressions_limit": 10,
"clicks_limit": 10,
"cost_per_impression": 1.2,
"cost_per_click": 2.9,
"ad_title": "Exclusive Access",
"ad_text": "Join us for exclusive access!",
"start_date": 0,
"end_date": 13,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "20024aab-2476-4d35-b96f-f47589f05474",
"campaign_data": {
"impressions_limit": 6,
"clicks_limit": 6,
"cost_per_impression": 1.8,
"cost_per_click": 2.5,
"ad_title": "Best Experience",
"ad_text": "Experience the best with us!",
"start_date": 0,
"end_date": 14,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "a68ef7d6-c50f-40d1-b70f-a32d7cdcd036",
"campaign_data": {
"impressions_limit": 5,
"clicks_limit": 5,
"cost_per_impression": 1.5,
"cost_per_click": 2.7,
"ad_title": "Incredible Quality",
"ad_text": "Get the quality you deserve!",
"start_date": 0,
"end_date": 12,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "960e2896-3f34-43a3-8aa1-d984bb92b96e",
"campaign_data": {
"impressions_limit": 7,
"clicks_limit": 7,
"cost_per_impression": 1.3,
"cost_per_click": 2.4,
"ad_title": "Best Value",
"ad_text": "Get the best value for your money!",
"start_date": 0,
"end_date": 11,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "80a7f6a9-5939-4d74-9a7e-e455b9fa0fdd",
"campaign_data": {
"impressions_limit": 6,
"clicks_limit": 6,
"cost_per_impression": 1.9,
"cost_per_click": 2.6,
"ad_title": "Exclusive Offer",
"ad_text": "Don't miss out on this offer!",
"start_date": 0,
"end_date": 15,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "6bb59511-360d-49dc-9626-be2f6b9b3a0a",
"campaign_data": {
"impressions_limit": 8,
"clicks_limit": 8,
"cost_per_impression": 1.7,
"cost_per_click": 2.3,
"ad_title": "Limited Time Deal",
"ad_text": "Act fast to get this deal!",
"start_date": 0,
"end_date": 13,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "d7a27dc9-7d8d-4f25-bea4-2528f88238d1",
"campaign_data": {
"impressions_limit": 9,
"clicks_limit": 9,
"cost_per_impression": 1.2,
"cost_per_click": 2.9,
"ad_title": "Discover Our Services",
"ad_text": "Find out what we can do for you!",
"start_date": 0,
"end_date": 14,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "e6cde283-ca4b-456c-b1fb-c2967918b5ef",
"campaign_data": {
"impressions_limit": 10,
"clicks_limit": 10,
"cost_per_impression": 1.5,
"cost_per_click": 2.8,
"ad_title": "Get Started Today",
"ad_text": "Join us and start saving!",
"start_date": 0,
"end_date": 12,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "2a51ba38-bed3-43af-bbb4-99093b72e0d1",
"campaign_data": {
"impressions_limit": 6,
"clicks_limit": 6,
"cost_per_impression": 1.8,
"cost_per_click": 2.5,
"ad_title": "Best Choice",
"ad_text": "Choose the best for your needs!",
"start_date": 0,
"end_date": 11,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "95a9a488-8fee-4aa6-9c21-458f8f5523ca",
"campaign_data": {
"impressions_limit": 7,
"clicks_limit": 7,
"cost_per_impression": 1.3,
"cost_per_click": 2.2,
"ad_title": "Exclusive Access",
"ad_text": "Get exclusive access to our services!",
"start_date": 0,
"end_date": 13,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "2fb992b8-4441-47f5-9176-1da68526eafb",
"campaign_data": {
"impressions_limit": 8,
"clicks_limit": 8,
"cost_per_impression": 1.9,
"cost_per_click": 2.6,
"ad_title": "Best Quality",
"ad_text": "Experience the best quality!",
"start_date": 0,
"end_date": 15,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "f0d160bd-18b4-4c88-917c-06b699abeeec",
"campaign_data": {
"impressions_limit": 9,
"clicks_limit": 9,
"cost_per_impression": 1.4,
"cost_per_click": 2.3,
"ad_title": "Incredible Quality",
"ad_text": "Get the quality you deserve!",
"start_date": 0,
"end_date": 12,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "d8147d8c-2bcd-4ac8-bab5-82e420509a7f",
"campaign_data": {
"impressions_limit": 10,
"clicks_limit": 10,
"cost_per_impression": 1.6,
"cost_per_click": 2.8,
"ad_title": "Discover Our Services",
"ad_text": "Find out what we can do for you!",
"start_date": 0,
"end_date": 14,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "94209720-2c10-4a55-bc37-c28838d08012",
"campaign_data": {
"impressions_limit": 6,
"clicks_limit": 6,
"cost_per_impression": 1.2,
"cost_per_click": 2.9,
"ad_title": "Best Value",
"ad_text": "Get the best value for your money!",
"start_date": 0,
"end_date": 11,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "33fae4ce-d252-4b34-af2b-c11e770f65b5",
"campaign_data": {
"impressions_limit": 7,
"clicks_limit": 7,
"cost_per_impression": 1.3,
"cost_per_click": 2.4,
"ad_title": "Exclusive Offer",
"ad_text": "Don't miss out on this offer!",
"start_date": 0,
"end_date": 12,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "2c0261ba-bd3c-4b19-a53c-44efa10819e5",
"campaign_data": {
"impressions_limit": 8,
"clicks_limit": 8,
"cost_per_impression": 1.9,
"cost_per_click": 2.6,
"ad_title": "Best Choice",
"ad_text": "Choose the best for your needs!",
"start_date": 0,
"end_date": 13,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "0892fbb7-fbc6-45e9-b52f-494258a9c8f8",
"campaign_data": {
"impressions_limit": 9,
"clicks_limit": 9,
"cost_per_impression": 1.5,
"cost_per_click": 2.3,
"ad_title": "Incredible Quality",
"ad_text": "Get the quality you deserve!",
"start_date": 0,
"end_date": 14,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "055af498-567a-4a4e-b25f-60a57b9b591c",
"campaign_data": {
"impressions_limit": 10,
"clicks_limit": 10,
"cost_per_impression": 1.6,
"cost_per_click": 2.8,
"ad_title": "Get Started Today",
"ad_text": "Join us and start saving!",
"start_date": 0,
"end_date": 12,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "0dfd7246-4cfa-4721-899b-e6dd9c483626",
"campaign_data": {
"impressions_limit": 6,
"clicks_limit": 6,
"cost_per_impression": 1.2,
"cost_per_click": 2.9,
"ad_title": "Best Experience",
"ad_text": "Experience the best with us!",
"start_date": 0,
"end_date": 11,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
},
{
"advertiser_id": "32d75bc6-0779-4081-98b3-58273497c561",
"campaign_data": {
"impressions_limit": 7,
"clicks_limit": 7,
"cost_per_impression": 1.3,
"cost_per_click": 2.2,
"ad_title": "Discover Our Services",
"ad_text": "Find out what we can do for you!",
"start_date": 0,
"end_date": 13,
"targeting": {
"gender": null,
"age_from": null,
"age_to": null,
"location": null
}
}
}
]
services/loadtest/mocks/ml_scores.json
+31352
View File
File diff suppressed because it is too large Load Diff
services/loadtest/static/index.html
+459
View File
@@ -0,0 +1,459 @@
<!DOCTYPE html>
<html lang="en" class="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AdNova Loadtest</title>
<script src="https://cdn.tailwindcss.com"></script>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<style>
body {
font-family: 'Inter', sans-serif;
}
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');
.chart-container {
position: relative;
height: 250px;
width: 100%;
}
.status-dot {
height: 10px;
width: 10px;
border-radius: 50%;
display: inline-block;
}
.tooltip {
visibility: hidden;
}
.has-tooltip:hover .tooltip {
visibility: visible;
}
</style>
</head>
<body class="bg-slate-900 text-gray-300">
<div class="container mx-auto p-4 md:p-8">
<header class="mb-8">
<h1 class="text-4xl font-bold text-white tracking-tight">AdNova Loadtest</h1>
</header>
<div class="grid grid-cols-1 lg:grid-cols-3 gap-8">
<!-- Left Column: Controls -->
<div class="lg:col-span-1 flex flex-col gap-6">
<!-- Mock Data Management -->
<div class="bg-slate-800 rounded-lg p-6 shadow-lg">
<h2 class="text-xl font-semibold text-white mb-4">1. Data Setup</h2>
<button id="loadMocksBtn"
class="w-full bg-indigo-600 hover:bg-indigo-500 text-white font-bold py-2 px-4 rounded-lg transition-colors duration-300 flex items-center justify-center">
Load Mock Data
</button>
<div id="mockStatus" class="mt-4 space-y-2 text-sm">
<div class="flex items-center justify-between"><span class="text-slate-300">Clients:</span><span
id="clientsStatus" class="font-mono">Checking...</span></div>
<div class="flex items-center justify-between"><span
class="text-slate-300">Advertisers:</span><span id="advertisersStatus"
class="font-mono">Checking...</span></div>
<div class="flex items-center justify-between"><span
class="text-slate-300">Campaigns:</span><span id="campaignsStatus"
class="font-mono">Checking...</span></div>
<div class="flex items-center justify-between"><span class="text-slate-300">ML
Scores:</span><span id="mlScoresStatus" class="font-mono">Checking...</span></div>
</div>
</div>
<!-- Test Configuration -->
<div id="config-panel" class="bg-slate-800 rounded-lg p-6 shadow-lg">
<h2 class="text-xl font-semibold text-white mb-4">2. Test Configuration</h2>
<div class="space-y-4">
<div>
<label for="loadProfile" class="block text-sm font-medium text-slate-300">Load
Profile</label>
<select id="loadProfile"
class="mt-1 block w-full bg-slate-700 border border-slate-600 rounded-md shadow-sm py-2 px-3 text-white focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm">
<option value="const">Constant</option>
<option value="line">Line</option>
<option value="step">Step</option>
<option value="once">Once</option>
<option value="unlimited">Unlimited</option>
</select>
</div>
<!-- Profile Specific Options -->
<div id="const-options">
<label for="maxRps" class="block text-sm font-medium text-slate-300">Requests Per Second
(RPS)</label>
<input type="number" id="maxRps" value="100"
class="mt-1 block w-full bg-slate-700 border border-slate-600 rounded-md shadow-sm py-2 px-3 text-white focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm">
</div>
<div id="line-options" class="hidden space-y-2">
<input type="number" id="fromRps" placeholder="From RPS (e.g., 10)"
class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
<input type="number" id="toRps" placeholder="To RPS (e.g., 100)"
class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
</div>
<div id="step-options" class="hidden space-y-2">
<input type="number" id="stepFromRps" placeholder="From RPS"
class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
<input type="number" id="stepToRps" placeholder="To RPS"
class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
<input type="number" id="stepRps" placeholder="Step Size (RPS)"
class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
<input type="number" id="stepDuration" placeholder="Step Duration (sec)"
class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
</div>
<div id="once-options" class="hidden">
<input type="number" id="onceCount" placeholder="Number of Requests"
class="block w-full bg-slate-700 border border-slate-600 rounded-md py-2 px-3 text-white">
</div>
<div id="unlimited-options"
class="hidden p-2 bg-yellow-900/50 rounded-md text-yellow-300 text-xs">
Warning: This profile sends requests as fast as possible and may exhaust system resources.
</div>
</div>
</div>
<!-- Start/Stop Controls -->
<div class="bg-slate-800 rounded-lg p-6 shadow-lg">
<h2 class="text-xl font-semibold text-white mb-4">3. Execute Test</h2>
<button id="startStopBtn"
class="w-full bg-green-600 hover:bg-green-500 text-white font-bold py-3 px-4 rounded-lg transition-colors duration-300 text-lg flex items-center justify-center">
Start Test
</button>
</div>
</div>
<!-- Right Column: Stats & Charts -->
<div class="lg:col-span-2 flex flex-col gap-6">
<!-- Summary Stats -->
<div class="grid grid-cols-1 md:grid-cols-3 gap-4">
<div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
<h3 class="text-sm font-medium text-slate-400">RPS</h3>
<p id="rpsStat" class="text-3xl font-semibold text-white">0</p>
</div>
<div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
<h3 class="text-sm font-medium text-slate-400">Avg Latency (ms)</h3>
<p id="latencyStat" class="text-3xl font-semibold text-white">0.00</p>
</div>
<div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
<h3 class="text-sm font-medium text-slate-400">Error Rate</h3>
<p id="errorRateStat" class="text-3xl font-semibold text-white">0.00%</p>
</div>
</div>
<div class="grid grid-cols-1 md:grid-cols-2 gap-4">
<div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
<h3 class="text-sm font-medium text-slate-400">Total Requests</h3>
<p id="totalReqsStat" class="text-2xl font-semibold text-white">0</p>
</div>
<div class="bg-slate-800 p-4 rounded-lg shadow-lg text-center">
<h3 class="text-sm font-medium text-slate-400">Total Errors</h3>
<p id="totalErrorsStat" class="text-2xl font-semibold text-white">0</p>
</div>
</div>
<!-- Charts -->
<div class="bg-slate-800 p-4 rounded-lg shadow-lg">
<h3 class="font-semibold text-white mb-2">Requests Per Second (RPS)</h3>
<div class="chart-container"><canvas id="rpsChart"></canvas></div>
</div>
<div class="bg-slate-800 p-4 rounded-lg shadow-lg">
<h3 class="font-semibold text-white mb-2">Average Latency (ms)</h3>
<div class="chart-container"><canvas id="latencyChart"></canvas></div>
</div>
<div class="bg-slate-800 p-4 rounded-lg shadow-lg">
<h3 class="font-semibold text-white mb-2">Error Rate (%)</h3>
<div class="chart-container"><canvas id="errorRateChart"></canvas></div>
</div>
</div>
</div>
</div>
<script>
const loadProfileSelect = document.getElementById( 'loadProfile' )
const startStopBtn = document.getElementById( 'startStopBtn' )
const loadMocksBtn = document.getElementById( 'loadMocksBtn' )
const configPanel = document.getElementById( 'config-panel' )
const rpsStat = document.getElementById( 'rpsStat' )
const latencyStat = document.getElementById( 'latencyStat' )
const errorRateStat = document.getElementById( 'errorRateStat' )
const totalReqsStat = document.getElementById( 'totalReqsStat' )
const totalErrorsStat = document.getElementById( 'totalErrorsStat' )
let ws
let rpsChart, latencyChart, errorRateChart
let isRunning = false
const MAX_DATA_POINTS = 60
function createChart ( ctx, label, color )
{
return new Chart( ctx, {
type: 'line',
data: {
labels: [],
datasets: [ {
label: label,
data: [],
borderColor: color,
backgroundColor: color + '33',
borderWidth: 2,
pointRadius: 0,
tension: 0.4,
fill: true,
} ]
},
options: {
responsive: true,
maintainAspectRatio: false,
scales: {
x: {
ticks: { color: '#94a3b8' },
grid: { color: '#334155' }
},
y: {
beginAtZero: true,
ticks: { color: '#94a3b8' },
grid: { color: '#334155' }
}
},
plugins: {
legend: { display: false }
}
}
} )
}
function updateChart ( chart, label, newData )
{
chart.data.labels.push( label )
chart.data.datasets[ 0 ].data.push( newData )
if ( chart.data.labels.length > MAX_DATA_POINTS )
{
chart.data.labels.shift()
chart.data.datasets[ 0 ].data.shift()
}
chart.update( 'none' )
}
function resetCharts ()
{
const charts = [ rpsChart, latencyChart, errorRateChart ]
charts.forEach( chart =>
{
if ( chart )
{
chart.data.labels = []
chart.data.datasets[ 0 ].data = []
chart.update()
}
} )
rpsStat.textContent = '0'
latencyStat.textContent = '0.00'
errorRateStat.textContent = '0.00%'
totalReqsStat.textContent = '0'
totalErrorsStat.textContent = '0'
}
function setupWebSocket ()
{
const protocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://'
const wsUrl = protocol + window.location.host + '/ws'
ws = new WebSocket( wsUrl )
ws.onopen = () => console.log( 'WebSocket connected' )
ws.onclose = () => console.log( 'WebSocket disconnected' )
ws.onerror = ( error ) => console.error( 'WebSocket error:', error )
ws.onmessage = ( event ) =>
{
const data = JSON.parse( event.data )
if ( typeof data.isRunning !== 'undefined' )
{
if ( data.isRunning )
{
isRunning = true
startStopBtn.textContent = 'Stop Test'
startStopBtn.classList.remove( 'bg-green-600', 'hover:bg-green-500' )
startStopBtn.classList.add( 'bg-red-600', 'hover:bg-red-500' )
configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = true )
}
}
if ( data.isRunning )
{
const now = new Date().toLocaleTimeString()
rpsStat.textContent = data.rps
latencyStat.textContent = data.latency.toFixed( 2 )
errorRateStat.textContent = data.errorRate.toFixed( 2 ) + '%'
totalReqsStat.textContent = data.totalReqs
totalErrorsStat.textContent = data.totalErrors
updateChart( rpsChart, now, data.rps )
updateChart( latencyChart, now, data.latency )
updateChart( errorRateChart, now, data.errorRate )
} else
{
totalReqsStat.textContent = data.totalReqs
totalErrorsStat.textContent = data.totalErrors
}
}
}
function setRunningState ()
{
isRunning = true
startStopBtn.textContent = 'Stop Test'
startStopBtn.classList.remove( 'bg-green-600', 'hover:bg-green-500' )
startStopBtn.classList.add( 'bg-red-600', 'hover:bg-red-500' )
configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = true )
resetCharts()
}
function setStoppedState ()
{
isRunning = false
startStopBtn.textContent = 'Start Test'
startStopBtn.classList.remove( 'bg-red-600', 'hover:bg-red-500' )
startStopBtn.classList.add( 'bg-green-600', 'hover:bg-green-500' )
configPanel.querySelectorAll( 'input, select' ).forEach( el => el.disabled = false )
}
startStopBtn.addEventListener( 'click', () =>
{
if ( isRunning )
{
fetch( '/api/stop-test', { method: 'POST' } )
.then( () => setStoppedState() )
} else
{
const config = {
loadProfile: loadProfileSelect.value,
maxRps: parseInt( document.getElementById( 'maxRps' ).value ) || 100,
fromRps: parseInt( document.getElementById( 'fromRps' ).value ) || 0,
toRps: parseInt( document.getElementById( 'toRps' ).value ) || 0,
stepFromRps: parseInt( document.getElementById( 'stepFromRps' ).value ) || 0,
stepToRps: parseInt( document.getElementById( 'stepToRps' ).value ) || 0,
stepRps: parseInt( document.getElementById( 'stepRps' ).value ) || 0,
stepDuration: parseInt( document.getElementById( 'stepDuration' ).value ) || 0,
onceCount: parseInt( document.getElementById( 'onceCount' ).value ) || 0,
}
if ( config.loadProfile === 'step' )
{
config.fromRps = config.stepFromRps
config.toRps = config.stepToRps
}
fetch( '/api/start-test', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify( config )
} ).then( res =>
{
if ( res.ok ) setRunningState()
else alert( "Failed to start test. Check console for details." )
} ).catch( err =>
{
console.error( "Error starting test:", err )
alert( "Failed to start test. Network error or server unreachable." )
} )
}
} )
function updateMockStatus ( status )
{
const statuses = {
clients: document.getElementById( 'clientsStatus' ),
advertisers: document.getElementById( 'advertisersStatus' ),
campaigns: document.getElementById( 'campaignsStatus' ),
ml_scores: document.getElementById( 'mlScoresStatus' ),
}
for ( const key in status )
{
const el = statuses[ key ]
if ( el )
{
if ( status[ key ] )
{
el.innerHTML = '<span class="status-dot bg-green-500 mr-2"></span>Loaded'
el.classList.remove( 'text-red-400' )
el.classList.add( 'text-green-400' )
} else
{
el.innerHTML = '<span class="status-dot bg-red-500 mr-2"></span>Not Loaded'
el.classList.remove( 'text-green-400' )
el.classList.add( 'text-red-400' )
}
}
}
}
function checkMocks ()
{
fetch( '/api/check-mocks' )
.then( res => res.json() )
.then( data => updateMockStatus( data ) )
.catch( err => console.error( "Failed to check mocks:", err ) )
}
loadMocksBtn.addEventListener( 'click', () =>
{
loadMocksBtn.textContent = 'Loading...'
loadMocksBtn.disabled = true
fetch( '/api/load-mocks', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify( {} )
} )
.then( res =>
{
if ( !res.ok ) alert( 'Failed to load mocks. Check console for details.' )
return res.json()
} )
.then( () =>
{
checkMocks()
} )
.finally( () =>
{
loadMocksBtn.textContent = 'Load Mock Data'
loadMocksBtn.disabled = false
} )
} )
loadProfileSelect.addEventListener( 'change', ( e ) =>
{
document.getElementById( 'const-options' ).classList.add( 'hidden' )
document.getElementById( 'line-options' ).classList.add( 'hidden' )
document.getElementById( 'step-options' ).classList.add( 'hidden' )
document.getElementById( 'once-options' ).classList.add( 'hidden' )
document.getElementById( 'unlimited-options' ).classList.add( 'hidden' )
document.getElementById( e.target.value + '-options' ).classList.remove( 'hidden' )
} )
window.onload = () =>
{
rpsChart = createChart( document.getElementById( 'rpsChart' ).getContext( '2d' ), 'RPS', '#6366f1' )
latencyChart = createChart( document.getElementById( 'latencyChart' ).getContext( '2d' ), 'Latency', '#34d399' )
errorRateChart = createChart( document.getElementById( 'errorRateChart' ).getContext( '2d' ), 'Error Rate', '#f87171' )
setupWebSocket()
checkMocks()
loadProfileSelect.dispatchEvent( new Event( 'change' ) )
};
</script>
</body>
</html>
services/telegram_bot/.env.template
+1 -1
View File
@@ -4,4 +4,4 @@
AIOGRAM_BOT_TOKEN=
AIOGRAM_BACKEND_ADDRESS=http://localhost:8080
REDIS_URI=redis://localhost:6379
MINIO_ENDPOINT=localhost:9000
MINIO_ENDPOINT=http://localhost:9000
services/telegram_bot/Dockerfile
+4 -4
View File
@@ -1,7 +1,7 @@
# Stage 1: Install dependencies
FROM docker.io/python:3.11-alpine3.20 AS builder
FROM docker.io/python:3.13-alpine3.22 AS builder
COPY --from=ghcr.io/astral-sh/uv:0.4.30 /uv /uvx /bin/
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
WORKDIR /app
@@ -17,7 +17,7 @@ RUN uv sync --no-dev --no-install-project --no-cache
# Stage 2: Start the application
FROM docker.io/python:3.11-alpine3.20
FROM docker.io/python:3.13-alpine3.22
WORKDIR /app
@@ -34,4 +34,4 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONOPTIMIZE=2 \
PATH="/opt/venv/bin:$PATH"
CMD python main.py
CMD [ "python", "main.py" ]
services/telegram_bot/README.md
+3 -3
View File
@@ -4,9 +4,9 @@
Ensure you have the following installed on your system:
- [Python](https://www.python.org/) (>=3.10,<3.12)
- [uv](https://docs.astral.sh/uv/)
- [Docker](https://www.docker.com/) (for containerized setup)
- [Python](https://www.python.org/) (>=3.10,<3.14)
- [uv](https://docs.astral.sh/uv/) (latest version recommended)
- [Docker](https://www.docker.com/) (for containerized setup, latest version recommended)
## Basic setup
services/telegram_bot/config.py
+1 -1
View File
@@ -13,4 +13,4 @@ API_ENDPOINT = os.getenv("AIOGRAM_BACKEND_URL", "http://localhost:8080")
REDIS_URI = os.getenv("REDIS_URI", "redis://localhost:6379")
MINIO_URL = f"http://{os.getenv('MINIO_ENDPOINT', 'localhost:9000')}"
MINIO_URL = os.getenv("MINIO_ENDPOINT", "http://localhost:9000")
services/telegram_bot/pyproject.toml
+35 -38
View File
@@ -1,22 +1,19 @@
[project]
name = "adnova-telegram_bot"
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.10,<3.12"
dependencies = [
"aiogram-dialog>=2.3.1",
"aiogram>=3.17.0",
"cachetools>=5.5.1",
"httpx>=0.28.1",
"openapi-python-client>=0.23.1",
"python-dotenv>=1.0.1",
"redis>=5.2.1",
"aiogram-dialog>=2.4.0,<3.0.0",
"aiogram>=3.17.0,<4.0.0",
"cachetools>=5.0.0,<6.0.0",
"httpx>=0.28.0,<0.29.0",
"python-dotenv>=1.1.0,<2.0.0",
"redis>=6.2.0,<7.0.0",
]
name = "adnova-telegram_bot"
readme = "README.md"
requires-python = ">=3.10,<3.14"
version = "0.1.0"
[dependency-groups]
dev = [
"ruff>=0.9.6",
]
dev = ["ruff"]
[tool.ruff]
builtins = []
@@ -49,7 +46,7 @@ extend-include = []
fix = false
fix-only = false
force-exclude = true
include = ["*.py", "*.pyi", "*.ipynb", "**/pyproject.toml"]
include = ["**/pyproject.toml", "*.ipynb", "*.py", "*.pyi"]
indent-width = 4
line-length = 79
namespace-packages = []
@@ -64,20 +61,20 @@ unsafe-fixes = false
[tool.ruff.analyze]
detect-string-imports = true
direction = "Dependencies"
exclude = []
include-dependencies = {}
preview = false
direction = "Dependencies"
exclude = []
include-dependencies = {}
preview = false
[tool.ruff.format]
docstring-code-format = true
docstring-code-format = true
docstring-code-line-length = 79
exclude = []
indent-style = "space"
line-ending = "lf"
preview = false
quote-style = "double"
skip-magic-trailing-comma = false
exclude = []
indent-style = "space"
line-ending = "lf"
preview = false
quote-style = "double"
skip-magic-trailing-comma = false
[tool.ruff.lint]
allowed-confusables = ["ℹ"]
@@ -92,23 +89,23 @@ extend-unsafe-fixes = []
external = []
fixable = ["ALL"]
ignore = [
"ARG",
"D",
"ANN401",
"COM812",
"DJ001",
"FBT001",
"FBT002",
"N813",
"PLR2004",
"RUF001",
"TC002",
"ANN401",
"ARG",
"COM812",
"D",
"DJ001",
"FBT001",
"FBT002",
"N813",
"PLR2004",
"RUF001",
"TC002",
]
logger-objects = []
per-file-ignores = {}
preview = false
select = ["ALL"]
task-tags = ["TODO", "FIXME", "HACK", "WORKOUT"]
task-tags = ["FIXME", "HACK", "TODO", "WORKOUT"]
typing-modules = []
unfixable = []
tests/README.md
+1 -1
View File
@@ -1,6 +1,6 @@
# AdNova Tests
There is `unit` and `e2e` tests available, unit tests are placed all around `backend` service folder and `e2e` tests placed [here](./e2e/).
There is `unit` and `e2e` tests available, `unit` tests are placed all around `backend` service folder and `e2e` tests placed [here](./e2e/).
## Running unit tests
tests/e2e/.env.template
+1 -1
View File
@@ -1,3 +1,3 @@
# Below all environment variables and default values
BACKEND_BASE_URL=http://127.0.0.1:8080
BACKEND_BASE_URL=http://127.0.0.1:13240
tests/e2e/README.md
+5 -5
View File
@@ -4,14 +4,14 @@
Ensure you have the following installed on your system:
- [Python](https://www.python.org/) (>=3.10,<3.12)
- [uv](https://docs.astral.sh/uv/)
- [Docker](https://www.docker.com/)
- [Docker compose](https://docs.docker.com/compose/) (latest versions)
- [Python](https://www.python.org/) (>=3.10,<3.14)
- [uv](https://docs.astral.sh/uv/) (latest version recommended)
- [Docker](https://www.docker.com/) (latest version recommended)
- [Docker compose](https://docs.docker.com/compose/) (latest version recommended)
## Warning
Please note that containers will use ports from 13241 to 13246 and 8080, so there is must be no listeners on this ports range.
Please note that containers will use ports from 13240 to 13248, so there is must be no listeners on this ports range.
## Setup
tests/e2e/conftest.py
+15 -1
View File
@@ -9,7 +9,7 @@ from dotenv import load_dotenv
load_dotenv()
BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:8080")
BACKEND_BASE_URL = os.getenv("BACKEND_BASE_URL", "http://127.0.0.1:13240")
@pytest.fixture(scope="session", autouse=True)
@@ -20,6 +20,16 @@ def docker_compose() -> Generator[None]:
args=[
"docker",
"compose",
"--profile",
"loadtest",
"--profile",
"minio",
"--profile",
"observability",
"--profile",
"proxy",
"--profile",
"telegram_bot",
"down",
],
check=True,
@@ -31,6 +41,8 @@ def docker_compose() -> Generator[None]:
"compose",
"--project-name",
"adnova-testing",
"--profile",
"minio",
"up",
"-d",
"--build",
@@ -50,6 +62,8 @@ def docker_compose() -> Generator[None]:
"compose",
"--project-name",
"adnova-testing",
"--profile",
"minio",
"down",
"-v",
],
tests/e2e/pyproject.toml
+36 -51
View File
@@ -1,18 +1,12 @@
[project]
name = "adnova-e2e-tests"
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.10,<3.12"
dependencies = [
"httpx>=0.28.1",
"pytest>=8.3.4",
"python-dotenv>=1.0.1",
]
dependencies = ["httpx>=0.28.1", "pytest>=8.3.4", "python-dotenv>=1.0.1"]
name = "adnova-tests-e2e"
readme = "README.md"
requires-python = ">=3.10,<3.14"
version = "0.1.0"
[dependency-groups]
dev = [
"ruff>=0.9.6",
]
dev = ["ruff"]
[tool.ruff]
builtins = []
@@ -45,7 +39,7 @@ extend-include = []
fix = false
fix-only = false
force-exclude = true
include = ["*.py", "*.pyi", "*.ipynb", "**/pyproject.toml"]
include = ["**/pyproject.toml", "*.ipynb", "*.py", "*.pyi"]
indent-width = 4
line-length = 79
namespace-packages = []
@@ -60,50 +54,41 @@ unsafe-fixes = false
[tool.ruff.analyze]
detect-string-imports = true
direction = "Dependencies"
exclude = []
include-dependencies = {}
preview = false
direction = "Dependencies"
exclude = []
include-dependencies = {}
preview = false
[tool.ruff.format]
docstring-code-format = true
docstring-code-format = true
docstring-code-line-length = 79
exclude = []
indent-style = "space"
line-ending = "lf"
preview = false
quote-style = "double"
skip-magic-trailing-comma = false
exclude = []
indent-style = "space"
line-ending = "lf"
preview = false
quote-style = "double"
skip-magic-trailing-comma = false
[tool.ruff.lint]
allowed-confusables = ["ℹ"]
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
exclude = ["tests.py"]
explicit-preview-rules = false
extend-fixable = []
allowed-confusables = ["ℹ"]
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
exclude = ["tests.py"]
explicit-preview-rules = false
extend-fixable = []
extend-per-file-ignores = {}
extend-safe-fixes = []
extend-select = []
extend-unsafe-fixes = []
external = []
fixable = ["ALL"]
ignore = [
"ARG",
"D",
"ANN401",
"COM812",
"FBT001",
"FBT002",
"N813",
"S101",
]
logger-objects = []
per-file-ignores = {}
preview = false
select = ["ALL"]
task-tags = ["TODO", "FIXME", "HACK", "WORKOUT"]
typing-modules = []
unfixable = []
extend-safe-fixes = []
extend-select = []
extend-unsafe-fixes = []
external = []
fixable = ["ALL"]
ignore = ["ANN401", "ARG", "COM812", "D", "FBT001", "FBT002", "N813", "S101"]
logger-objects = []
per-file-ignores = {}
preview = false
select = ["ALL"]
task-tags = ["FIXME", "HACK", "TODO", "WORKOUT"]
typing-modules = []
unfixable = []
[tool.ruff.lint.pylint]
max-args = 6