Private deployment topologies

In one paragraph

A private deployment runs models and the systems around them inside an environment the client controls: their own building, their own cloud account, or a network with no outside connection at all. The topology is chosen from where the data may sit, what hardware already exists, how steady the load is, and who will operate it.

How it works

Common to all: a model gateway in front of every model, identity from the client's own provider, private networking, egress allowed only to listed hosts, and a data-residency map covering primary stores, backups, logs, indexes and vendor support access.

Design choices and trade-offs

• Own or rent. Owning hardware pays when load is steady and high; renting suits variable or uncertain load. The decision is made from measured load, not assumed.
• Kubernetes or plain servers. Kubernetes helps with several models and autoscaling, using the vendor's GPU device plugin and Dynamic Resource Allocation; plain servers are easier for a small team to run.
• Where weights come from. Pulled from an internal registry, never directly from the internet at runtime.

When not to use it

When the data is not sensitive, volumes are low, and a hosted service's terms are acceptable, a hosted API may be the simpler choice.

How we evaluate it

Capacity tests at the expected concurrency on the target hardware, restore tests of backups, egress tests confirming nothing reaches unlisted hosts, and failure drills for the components named in each reference deployment.

Sources

• Kubernetes Dynamic Resource Allocation (opens in a new tab)
• Explainers: Private deployment, Owning or renting the hardware