On-Prem and Air-Gapped LLM Deployment With vLLM and Ollama: What FDEs Need for Regulated Clients
A regulated client may approve an AI use case and still reject every managed model API. Their documents cannot leave the site, their production network has no outbound internet path and a GPU server cannot install packages from a public repository. An on premise LLM deployment, more commonly written as on-premises LLM deployment, changes the FDE's job from calling a model endpoint to operating a secure inference service with a controlled software and model supply chain.
This is not a policy guide or a recommendation that every client self-host a model. It is an infrastructure field guide for deciding when the constraint is real, choosing between vLLM and Ollama, sizing and staging a deployment, and proving that the result is usable and supportable. In regulated work, the first successful response is the beginning of the engagement, not the end.
What does on premise LLM deployment mean for an FDE?
On premise LLM deployment runs model inference on infrastructure the customer operates or controls, rather than sending prompts to a public hosted model endpoint. An air-gapped deployment is a stricter form: the runtime environment has no direct internet connection, so software, model weights, updates and support artifacts move through an approved offline transfer process.
The term does not automatically mean safer, compliant, cheaper or private. A self-hosted model can still leak data through an over-broad retrieval index, an exposed API, permissive logs or a copied USB drive. Likewise, an on-prem server might have restricted but approved outbound connectivity and therefore not be air-gapped. Establish the actual boundary before promising an architecture.
For an FDE, the work spans hardware, drivers, containers, model artifacts, networking, identity, application integration, observability, operations and customer handoff. That is why infrastructure belongs in the complete FDE tech stack, not only on a platform team's diagram.
SCALER | 12 MONTH PROGRAM
AI Forward Deployed
Engineer Program
Full-stack engineering, production AI, and client-facing consulting — built around a real client engagement you run from first call to production scale.
ENROLL NOWHiring from this program
vLLM vs Ollama: which runtime should an FDE choose?
vLLM and Ollama overlap, but they solve different parts of the operating problem. Treat them as options to validate against the target model, accelerator, load profile and support model, not as a universal production ranking.
vLLM's quickstart documents offline batch inference and online serving across several accelerator backends. It is commonly a better fit when you need a high-throughput, API-oriented inference service, concurrency controls and an OpenAI-compatible serving surface for an application or gateway. Its installation and model compatibility must be tested against the exact GPU, driver, container image and model configuration you plan to ship.
Ollama's Linux documentation describes a local service and manual installation path, while its import guide supports building a model from local Safetensors weights or a prepared GGUF file with a Modelfile. That makes it practical for local development, evaluation and some controlled small-scale serving patterns. It is not a shortcut around production design: expose it behind an authenticated gateway, use a dedicated service identity and plan its model, version and update lifecycle.
| Decision factor | Favour vLLM | Favour Ollama | Question to answer before choosing |
|---|---|---|---|
| Primary job | Shared inference API for an application or multiple internal consumers | Local developer workflow, evaluation box or a simpler controlled service | Is this a multi-user production endpoint or a contained local workflow? |
| Serving model | Online serving and offline batch inference with an application-facing API | Local model management and a local API | Which runtime supports the exact model format and acceleration path? |
| Scale concern | Throughput, concurrency and GPU utilisation need active benchmarking | Small concurrency and operational simplicity matter most | What are the target requests, tokens, context lengths and latency percentile? |
| Model artifact path | Approved local model directory or staged cache mounted into a runtime | Locally imported Safetensors or prepared GGUF through a Modelfile | Can all weights, tokenizers and configs be staged without a runtime pull? |
| Operating model | Platform or application team can run a hardened service, gateway and monitoring | Team needs a narrow service with a clearly bounded workload | Who patches, rotates credentials, reads logs and handles an outage at 02:00? |
A useful default is: select a small approved model, build a repeatable air-gap artifact process, benchmark a representative workload, then choose the runtime that meets the measured requirement. Do not make a model-server decision from a laptop demo or a vendor benchmark using another context length and GPU generation.
What does a secure on-prem LLM architecture look like?
Separate the user-facing application from the inference runtime. The model server should not receive a user's broad enterprise credential, connect directly to every internal system or expose an unauthenticated port to the whole network.
The model endpoint also needs its own access control. For a narrow proof of value, bind the runtime only to a private interface or service network, place a reverse proxy or API gateway in front, require authentication and allow only known callers. The NVIDIA Container Toolkit installation guidance shows that containerised GPU workloads depend on a supported container engine, the toolkit and compatible driver setup. In a regulated environment, pin and approve those components rather than installing whatever the public package manager offers that day.
Build an AI-First Career, Master the Complete Skillset
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
AI Forward Deployed Engineer Program
Full-stack engineering, production AI and client-facing consulting
+1000 moreHow should you size GPU, memory and capacity?
Do not begin with a GPU shopping list. Capacity is a workload question, and the same model can behave very differently with a longer context window, a larger output cap, more concurrent users, a different quantisation or tool-call workload.
Build a sizing worksheet before procurement or deployment:
| Input to measure | Why it changes the design | Example acceptance decision |
|---|---|---|
| Selected model, revision and licence | Determines weights, tokenizer, runtime support and legal use constraints | Artifact is approved, immutable and reproducible from the manifest |
| Precision or quantisation | Changes memory use, quality and compatible runtime path | Quality evaluation remains above the agreed threshold at the selected format |
| Input context and output limits | Drives KV-cache memory and latency | Normal request and long-document request both stay within the service objective |
| Peak concurrent requests | Determines queueing, batching and required serving capacity | P95 latency remains within target at expected simultaneous use |
| Input and output tokens per request | More useful than request count alone for LLM load | Benchmark reflects the customer's real prompt and answer distribution |
| Availability target | Determines whether you need one host, spare capacity or a failover design | Planned maintenance and host failure behaviour are documented and accepted |
| Retrieval and application load | Uses CPU, RAM, storage and network even when model inference is on GPU | Retrieval does not become the bottleneck or bypass data filters |
Memory planning has at least three buckets: model weights, runtime overhead and the KV cache used for active contexts. The weight footprint alone is therefore not a capacity plan. Start with a small, representative evaluation set and traffic replay. Measure time to first token, tokens per second, P50 and P95 end-to-end latency, GPU memory use, queue depth, error rate and user-rated usefulness. Repeat under the context limits and concurrency you will actually permit.
Keep a safety margin. A service that works only while the GPU is nearly full has little room for a longer-than-usual document, a concurrent user burst or maintenance. If the client cannot state a volume target yet, deploy a capped pilot with a queue and usage telemetry rather than presenting a speculative server size as a fact.
How do you move models and software into an air-gapped environment?
An air gap converts downloading into release engineering. The safest mental model is a controlled artifact pipeline: acquire in a connected staging zone, verify and scan, record provenance, transfer through an approved channel, validate again inside, then deploy from an internal approved repository or mounted volume.
Build an offline release manifest
The manifest is the source of truth for one release. Include the runtime and container image digests, operating-system packages, GPU driver and toolkit versions, model weights and tokenizer hashes, model card or licence reference, configuration files, SBOM where available, signatures, test results and rollback version. A filename such as latest-model is not a release identifier.
Stage and verify outside the air gap
Use a connected build or staging zone approved by the customer. Download the exact artifacts, resolve dependencies, scan them according to the client's process and calculate checksums. Capture package indexes or container images locally rather than allowing the production host to resolve dependencies from the internet. Have the model owner confirm that the model and licence are suitable for the intended use before transfer.
Transfer, verify and import inside
Use the client's approved media or one-way transfer route. Verify hashes against the signed manifest after transfer and record chain-of-custody evidence. Keep the runtime host unable to make outbound DNS or HTTPS connections unless the design explicitly allows an approved exception.
For Ollama, its official import flow can create a local model from a directory of Safetensors weights or a prepared GGUF file, referenced from a Modelfile, then built with ollama create. That is materially different from relying on ollama pull in the restricted environment. For vLLM, stage the exact model directory and dependent artifacts into an approved local path or image volume, then configure the server to use that local artifact. Test this before the customer acceptance window, not after disconnecting the network.
Make updates a routine, not an emergency improvisation
Every driver patch, model change, tokenizer change and container update follows the same offline release path, test gate and rollback procedure. Maintain a vulnerability and support watch in the connected zone, because an air-gapped runtime cannot tell you that an upstream advisory exists. The customer must decide the patch cadence and exception process; the FDE makes them executable.
Which controls are non-negotiable for a first regulated deployment?
The exact control set is client-specific, but the following baseline prevents common design failures:
- Deny-by-default egress. Verify at host and network layers that the runtime cannot silently reach public model, package or telemetry endpoints.
- Immutable approved artifacts. Pin runtime images, package versions, models, tokenizers and configurations; record hashes and provenance.
- Separate identities. Use distinct identities for deployment automation, application-to-model calls, retrieval services and human operators.
- Least-privilege network paths. Permit only the application gateway, internal monitoring and approved administrative route. Do not expose port 11434, 8000 or any runtime port broadly by default.
- User-level data authorisation. Apply access filters before retrieval context reaches the model. A private model does not make over-broad retrieval acceptable.
- Secrets outside prompts and images. Use the customer's approved secret store or protected configuration path, rotate credentials and redact them from logs.
- Auditable, privacy-aware telemetry. Log request identifiers, version, latency, errors, retrieval or tool decisions and approval events without indiscriminately storing prompts or sensitive source text.
- Read-only first release. If the assistant can affect a system of record, start with drafts and require deterministic validation and human approval for writes.
- Break-glass and rollback. Define who can disable the service, revoke access, revert a model and preserve evidence after an incident.
- Operational ownership. Name the team responsible for host patching, GPU health, capacity, certificates, model releases and user support after handoff.
This is security engineering applied to a deployed product. For a broader view of the languages and frameworks around these systems, see the tools, frameworks and languages FDEs use.
How do you test an air-gapped LLM deployment before launch?
A green health endpoint proves only that a process started. Acceptance should include functionality, boundary controls, recoverability and operations.
| Test category | Test | Passing evidence |
|---|---|---|
| Isolation | Attempt approved and unapproved egress from the runtime network | Approved internal paths work; unapproved DNS and internet routes are blocked and recorded |
| Reproducibility | Rebuild or restore the service from the offline manifest | Same pinned versions, hashes and configuration are deployed without external downloads |
| Identity | Call the gateway with valid, expired and wrong-scope identities | Only permitted callers reach the model or retrieval service; failures are attributable |
| Data boundary | Query with two mock users that have different document rights | Each result contains only authorised, cited sources |
| Model safety | Run injection, no-answer, stale-source and sensitive-data cases | Controls block forbidden actions and the system responds safely when evidence is absent |
| Performance | Replay representative prompts at expected concurrency | Measured latency, queueing and error rate meet the agreed pilot objective |
| Recovery | Restart a runtime, revoke a caller and roll back one model release | Service recovery and rollback follow the runbook and leave an audit trail |
| Handoff | Ask the operations team to use the runbook without the FDE | They can check health, find logs, triage a known fault and escalate correctly |
Include a test for accidental network dependence. Disable the staging proxy or remove all external routes, restart the runtime and run an evaluation set. If the server tries to fetch a tokenizer, model revision or kernel at startup, you have found a release-process gap. vLLM's documentation notes that some precompiled kernels can otherwise download and compile at startup on certain GPU generations, which is precisely the kind of dependency an offline release needs to pre-stage.
How Scaler Transformed Careers in Different Fields
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreWhat should an FDE do in the first two weeks?
Week 1: confirm the constraint and the operating boundary
- Identify the user workflow, sensitivity of the inputs, systems of record and a low-risk read-only first outcome.
- Clarify whether the environment is on-premises, restricted-egress or truly air-gapped. Document who approves exceptions.
- Inventory hardware, GPU type and memory, driver baseline, OS, container engine, network zones, identity provider, internal repositories and operations owners.
- Define a release manifest and offline transfer process before selecting an unverified model from a public catalogue.
- Create an acceptance plan covering quality, data entitlements, isolation, latency, monitoring and rollback.
Week 2: prove one thin, controlled path
- Stage one approved model and runtime into a non-production restricted environment and verify the manifest end to end.
- Serve a private authenticated endpoint, then connect one small application with safe sample data and user-level retrieval filters.
- Run functional, adversarial, egress, performance and recovery tests. Record the actual baseline, limits and failures.
- Produce an architecture note, threat model, capacity hypothesis, runbook and gated next-step recommendation. Do not promote to production because the first prompt succeeded.
This process needs the technical depth, calm debugging and stakeholder translation covered in the core skills of an FDE. Your client needs both a working server and an understandable path to own it.
When should you not deploy vLLM or Ollama on-prem?
Do not self-host simply because a model is available. A managed offering or a customer VPC may be the better answer when data processing is approved externally, demand is spiky, the organisation lacks GPU operations capacity or model availability must scale beyond a small local fleet. Self-hosting can add expensive capacity, patch and incident obligations without reducing the actual business risk.
Do not force an air gap when the customer has a well-governed private connectivity pattern that meets the real requirement. Conversely, do not interpret a broad "no external data" instruction as permission to copy weights and customer documents to an unmanaged workstation. The deployment model must match the verified constraint and the team that will operate it.
For a career path that connects this technical work to customer delivery, use a step-by-step roadmap to become an FDE.
Want structured practice building and deploying enterprise AI systems? Scaler's AI Forward Deployed Engineer course covers software engineering, LLM applications, evaluations, integrations and delivery for real-world AI deployments.
FAQs
Turn Learning into Career Growth
What is an on-prem LLM deployment?
An on-prem LLM deployment runs model inference on infrastructure the customer controls, such as its data centre, site or private environment, rather than a public model endpoint. It can support data-locality or network constraints, but it still needs identity controls, secure retrieval, model lifecycle management, patching, monitoring and operational ownership to be trustworthy.
Is an air-gapped LLM deployment completely offline?
The runtime environment has no direct internet access, but it is not maintenance-free. Models, drivers, containers, packages and security updates still require a controlled offline release process. Teams stage, scan, verify and transfer approved artifacts through an authorised path, then validate hashes and functionality inside the restricted environment before promotion.
Should I use vLLM or Ollama for on-prem inference?
Choose based on the model, hardware, concurrency, API needs and operating team. vLLM often suits application-facing high-throughput serving and benchmarking. Ollama can be useful for local development, evaluations and bounded local services, including imported local artifacts. Benchmark the precise model and workload before standardising on either runtime.
How do you size a GPU server for a private LLM?
Measure the approved model, precision, input context, output cap, concurrent users, tokens per request, latency target and availability requirement. Account for model weights, runtime overhead and KV cache, not weights alone. Then benchmark representative traffic and retain a safety margin. A laptop prompt test cannot establish production GPU capacity or service-level behaviour.
Can Ollama run in an air-gapped environment?
Yes, if you stage its binary and model artifacts through the approved offline process. Ollama documents creating a local model from Safetensors weights or a prepared GGUF file using a Modelfile and ollama create. Do not depend on a runtime pull from a public registry once the restricted environment is isolated.
What security controls does an on-prem LLM need?
Start with deny-by-default egress, pinned and verified artifacts, least-privilege identities, restricted network paths, user-level retrieval authorisation, protected secrets, audit-ready but privacy-aware logs, human approval for consequential writes and a tested rollback path. The exact controls depend on the customer, data classification and workflow, so validate them through security and operations review.
Do air-gapped LLMs need monitoring and updates?
Yes. Monitoring should remain inside the approved network and track availability, latency, GPU health, errors, queueing, version and security-relevant events without exposing sensitive prompts unnecessarily. Updates require the same signed, scanned and verified artifact process used for the original release, plus testing and a documented rollback before production promotion.
