Skip to content

Architecture

This page describes the internal architecture of proxmox-sdk — covering the package layers, the dual-mode FastAPI server, the standalone SDK, and the key design decisions behind each component.

Package Layers

The project is organized into five distinct layers, each independently usable:

flowchart TD
    CLI["proxmox_cli/\nTyper CLI + Textual TUI\n(interactive use)"]
    SERVER["proxmox_sdk/main.py\nDual-mode FastAPI server\n(mock OR real API proxy)"]
    CODEGEN["proxmox_codegen/\nPlaywright crawler → OpenAPI + Pydantic\n(schema generation pipeline)"]
    SDK["sdk/\nStandalone async Python SDK\n(production integrations)"]
    MOCK["mock/\nIn-memory CRUD state store\n(test infrastructure)"]
    GEN["generated/proxmox/\nPre-built OpenAPI schemas\n+ Pydantic models"]

    CLI -->|uses| SDK
    CLI -->|may use| SERVER
    SERVER -->|mock mode uses| MOCK
    SERVER -->|real mode uses| SDK
    CODEGEN -->|produces| GEN
    MOCK -->|loaded from| GEN
    SDK -->|mock backend loads| GEN
Layer Entry point Standalone? Description
sdk/ proxmox_sdk.ProxmoxSDK Yes Async + sync Python SDK with 5 backends
mock/ proxmox_sdk.mock_main Yes FastAPI mock server only
proxmox_sdk/main.py proxmox_sdk.main Yes Dual-mode FastAPI server
proxmox_codegen/ proxmox-sdk-codegen CLI Yes Schema generation pipeline
proxmox_cli/ proxmox / pbx CLI Yes Typer CLI and Textual TUI

Dual-Mode FastAPI Server

The FastAPI server can run in two modes, selected via PROXMOX_API_MODE:

Mock mode loads pre-generated OpenAPI schemas plus route metadata and creates CRUD endpoints backed by configurable mock-state stores. SQLite/WAL is the default. No real Proxmox server required.

flowchart TD
    REQ["HTTP Request"]
    ROUTER["FastAPI Router"]
    MOCK_EP["Generated Mock Dispatcher\n(from route metadata)"]
    MODELS["Lazy Pydantic Shards\nmodels/<group>.py"]
    STATE["SQLiteMockStore\nWAL row store"]
    RESP["JSON Response"]

    REQ --> ROUTER
    ROUTER --> MOCK_EP
    MOCK_EP -->|"load model when needed"| MODELS
    MOCK_EP -->|"GET: read"| STATE
    MOCK_EP -->|"POST/PUT: write"| STATE
    MOCK_EP -->|"DELETE: remove"| STATE
    STATE --> RESP

Startup flow:

  1. Load generated/proxmox/latest/openapi.json
  2. Load generated/proxmox/latest/route_metadata.json
  3. register_generated_proxmox_mock_routes() iterates 675 metadata operations across 449 paths
  4. For each operation, create a lightweight FastAPI dispatcher with precomputed path, parameter, and mock-topology metadata
  5. Load request/response model classes lazily from models/<group>.py through model_index.json
  6. Routes perform CRUD on SQLiteMockStore by default, with shared-memory and dict backends available by env var
  7. State is scoped by namespace and schema fingerprint; the default tempdir SQLite database is not a production persistence layer

Performance: ~2.1 s full app import in local smoke tests, ~0.75 s generated route registration, ~4 ms warm SQLite/WAL mock request latency via local TestClient smoke tests.

Real mode proxies requests to an actual Proxmox VE API, validating both the incoming request and the Proxmox response against the OpenAPI schema.

flowchart TD
    REQ["HTTP Request"]
    ROUTER["FastAPI Router"]
    REAL_EP["Real API Endpoint\n(validates request)"]
    CLIENT["ProxmoxClient\n(HttpsBackend)"]
    PVE["Real Proxmox VE\n:8006"]
    VALID["Response Validation\n(Pydantic)"]
    RESP["JSON Response"]

    REQ --> ROUTER
    ROUTER --> REAL_EP
    REAL_EP --> CLIENT
    CLIENT -->|"aiohttp HTTPS"| PVE
    PVE -->|"JSON data"| CLIENT
    CLIENT --> VALID
    VALID --> RESP

Startup flow:

  1. Load ProxmoxConfig from environment variables
  2. register_proxmox_routes() creates endpoints mirroring the mock structure
  3. Each request: validate → call ProxmoxClient.request() → validate response → return

Performance: ~500 ms startup, Proxmox latency + ~20–100 ms validation overhead.

SDK Architecture

The standalone SDK (sdk/) is the primary way for Python applications to query Proxmox programmatically without running the FastAPI server.

flowchart TD
    USER["Application code\n(proxbox-api, CLI, tests)"]
    SDK_CLASS["ProxmoxSDK\nsdk/api.py"]
    ROOT["ProxmoxResource('_/api2/json')\nsdk/resource.py"]
    NAV["Navigation chain\n.nodes('pve1').qemu(100).config"]
    LEAF["ProxmoxResource\npath=/api2/json/nodes/pve1/qemu/100/config"]
    BACKEND["AbstractBackend\nsdk/backends/base.py"]
    HTTPS["HttpsBackend\naiohttp + auth"]
    MOCK_B["MockBackend\nin-memory"]
    SSH["SshParamikoBackend\nor OpenSshBackend"]
    LOCAL["LocalBackend\npvesh subprocess"]
    PROXMOX["Proxmox VE"]

    USER -->|"ProxmoxSDK(host=..., user=..., ...)"| SDK_CLASS
    SDK_CLASS -->|"creates"| ROOT
    ROOT -->|"__getattr__ / __call__"| NAV
    NAV -->|"each step creates new resource"| LEAF
    LEAF -->|".get() / .post() / ..."| BACKEND
    BACKEND -->|"https"| HTTPS
    BACKEND -->|"mock"| MOCK_B
    BACKEND -->|"ssh_paramiko / openssh"| SSH
    BACKEND -->|"local"| LOCAL
    HTTPS -->|"aiohttp HTTPS"| PROXMOX
    MOCK_B -->|"in-memory"| PROXMOX
    SSH -->|"pvesh via SSH"| PROXMOX
    LOCAL -->|"pvesh subprocess"| PROXMOX

Code Generation Pipeline

The codegen pipeline converts the Proxmox VE API Viewer into reusable artifacts:

flowchart LR
    PVE["Proxmox VE\nAPI Viewer"]
    CRAWL["ProxmoxCrawler\nPlaywright"]
    PARSE["apidoc_parser.py"]
    NORM["normalize.py"]
    OA["OpenAPIBuilder\n→ openapi.json\n5.2 MB · 675 ops"]
    PYD["PydanticBuilder\n→ pydantic_models.py\n+ model shards"]
    META["RouteMetadataBuilder\n→ route_metadata.json\n+ model_index.json"]
    USE1["proxmox-sdk\nMock routes"]
    USE2["proxbox-api\nResponse validation"]

    PVE --> CRAWL
    CRAWL --> PARSE
    PARSE --> NORM
    NORM --> OA
    OA --> PYD
    OA --> META
    OA --> USE1
    META --> USE1
    PYD --> USE2

See Code Generation Pipeline for the full stage-by-stage breakdown.

Key Design Decisions

1. Dual-Mode Architecture

A single codebase serves both development (mock) and production (real proxy) use cases. The same API surface — same paths, same schemas, same validation — is available in both modes. Switching from mock to real is a single environment variable change.

Mock Real
Proxmox server required No Yes
Data persistence SQLite/WAL mock state by default Real Proxmox state
Request validation Yes (Pydantic) Yes (Pydantic)
Response validation Yes (Pydantic) Yes (Pydantic)
Startup time ~2.1 s full app import, ~0.75 s generated route registration ~500 ms + route registration

2. Metadata-Driven Route Registration

Rather than hardcoding 675 routes or rebuilding all route topology at runtime, the codegen pipeline emits route_metadata.json, model_index.json, and route-group model shards under each generated/proxmox/<version>/ directory. The server registers lightweight FastAPI dispatchers from this metadata and lazy-loads the matching Pydantic shard on the first request that needs request/response validation. The generated OpenAPI document is merged into the app docs directly, so Swagger/ReDoc stay complete without importing every model at startup.

3. Pre-Generated Schemas

The OpenAPI schema, aggregate Pydantic models, lazy model shards, model index, and route metadata are committed to the repository so mock mode works completely offline without a Proxmox server. Each version (e.g., 9.1.11) is stored independently under generated/proxmox/; CI ships both 9.1.11/ and a latest mirror.

4. Full Request/Response Validation

Every request body and response passes through Pydantic v2 validation against the OpenAPI-derived models. This enforces type correctness, produces accurate Swagger UI documentation, and catches schema drift between the SDK and Proxmox API.

5. SQLite Mock State

Mock state uses SQLiteMockStore by default. It stores objects, collection members, metadata, and tombstones as separate rows in a tempdir-scoped SQLite database using WAL mode, which avoids whole-state JSON serialization on every request. SharedMemoryMockStore and DictMockStore remain available through PROXMOX_MOCK_STORE=shared-memory|dict for compatibility and specialized test cases. Value blobs use orjson when installed, with a stdlib JSON fallback.

6. aiohttp for HTTPS Transport

The HTTPS backend uses aiohttp for its native async/await support, mature session pooling, and robust SSL/TLS handling. The session is created lazily on first request and reused across all subsequent calls.

Security Model

Mock Mode

  • No authentication by default — safe for development environments
  • Rate limiting is applied only on codegen endpoints via SlowAPI (hardcoded in rate_limit.py)
  • Codegen endpoints require CODEGEN_API_KEY Bearer token

Real Mode

  • All outbound Proxmox calls use API token auth (recommended) or password/ticket auth
  • SSL/TLS verification configurable via PROXMOX_API_VERIFY_SSL
  • No credentials stored in memory after session initialization
  • SensitiveDataFilter redacts passwords and tokens from all log output

See Security for the full reference.

Performance Characteristics

Metric Mock Mode Real Mode
Startup time ~2.1 s full app import Proxmox latency + route registration
Request latency ~4 ms local mock GET smoke Proxmox latency + ~10–20 ms validation overhead
Memory footprint ~100 MB ~80 MB
Max throughput 10,000+ req/s (FastAPI-bound) Proxmox server capacity

Key optimizations:

  • Lazy package imports — import proxmox_sdk does not construct any FastAPI app
  • Generated route metadata — startup loads route metadata instead of rebuilding topology and signatures
  • Lazy model shards — Pydantic route-group shards load only when a request needs validation
  • SQLite/WAL mock state — mock CRUD stores rows per object/member instead of serializing one whole-state blob
  • Cached URL components — HttpsBackend caches (scheme, netloc, base_path) to skip per-request URL parsing
  • Schema fingerprint caching — ProxmoxSchemaValue.fingerprint is a @cached_property

See Performance for the complete optimization reference.

Extension Points

  1. Custom mock data — Point PROXMOX_MOCK_DATA_PATH at a JSON/YAML file with initial mock state
  2. New Proxmox version — Run the codegen pipeline, store artifacts under generated/proxmox/<version>/
  3. Custom middleware — Add FastAPI middleware for auth, logging, or rate limiting in main.py
  4. New SDK backend — Implement AbstractBackend.request() and AbstractBackend.close(), register in _create_backend()

Testing Strategy

Test type Scope File
Unit Schema loading and validation test_schema.py
Unit Individual CRUD operations test_mock_routes.py
Unit Client auth and request logic test_proxmox_client.py (mocked)
Integration Full app startup in both modes test_main_app.py
Integration Custom mock data loading test_custom_mock_data.py
E2E Manual via Swagger UI —
E2E Real Proxmox environment Requires live cluster