Datarush/ARCHITECTURE.md

Architecture and Technologies

Technologies Used

PostgreSQL

PostgreSQL was selected as the primary database for its reliability, ACID compliance, and advanced features like JSONB support, which are useful for handling semi-structured data.

Redis

Redis is used for caching and session management due to its in-memory data store capabilities, which provide extremely low-latency data access.

MinIO

MinIO serves as the object storage solution, offering S3-compatible APIs. It is lightweight, performant, and ideal for storing large amounts of unstructured data.

Docker and Docker Compose

Docker is used to containerize the application, ensuring consistency across development and production environments. Docker Compose simplifies the orchestration of multiple services during development.

Nginx

Nginx is used as a reverse proxy and load balancer to distribute traffic efficiently and ensure high availability.

Makefile

The Makefile is used to automate common tasks such as building, testing, and running the application, improving developer productivity.

Services Architecture

Auth Service

• Responsibilities: Handles user authentication and authorization.
• Endpoints: Defined in auth.proto.

User Service

• Responsibilities: Manages user profiles and related data.
• Endpoints: Defined in user.proto.

Competition Service

• Responsibilities: Manages competitions, including creation, updates, and participant management.
• Endpoints: Defined in competition.proto.

Task Service

• Responsibilities: Handles tasks within competitions, including creation and updates.
• Endpoints: Defined in task.proto.

Submission Service

• Responsibilities: Handles submissions made by users for tasks.
• Endpoints: Defined in submission.proto.
• Additional Logic: Sends messages to Kafka for downstream processing.

Checker Adapter Service

• Responsibilities: Processes new submissions by running checks and validations.
• Additional Logic:
  • Reads new submission messages from Kafka.
  • Runs Kubernetes pods to execute the checker logic.
  • Retrieves results and updates the database.

Results Service

• Responsibilities: Manages and retrieves results for tasks and competitions.
• Endpoints: Defined in results.proto.

Review Service

• Responsibilities: Handles reviews for submissions.
• Endpoints: Defined in review.proto.

Achievements Service

• Responsibilities: Manages user achievements and badges.
• Endpoints: Defined in achievements.proto.

Communication and Workflow

• Kafka: Acts as the message broker for asynchronous communication between the Submission Service and the Checker Adapter Service.
• Kubernetes: Used by the Checker Adapter Service to dynamically run pods for executing submission checks.
• PostgreSQL: Centralized database for storing results and other persistent data.