# AWS deployment and production operations

The draft requires integration with the Library's AWS systems and standard import/export without proprietary lock-in. The components below are a proposed implementation, not a mandated stack. Confirm available services, network boundaries, security rules and delivery interfaces with the Library. [SOW §3.7](https://sam.gov/api/prod/opps/v3/opportunities/resources/files/34eacfe92b1a46228b9f90674c4c60e1/download).

## How deployment responsibilities connect

<!-- diagram:deployment -->

## Proposed deployment components

| Component | Candidate implementation | Responsibility |
|---|---|---|
| Control plane | Typed application service | Book versions, jobs, profiles, approvals and audit events. |
| Metadata and job state | PostgreSQL | Structured records, transactions and searchable status. |
| Artifact storage | S3 | Separate originals, intermediates, staging and accepted outputs. |
| Speech and analysis workers | Containerized Python workers with pinned models | GPU inference or appropriate CPU inference. |
| Compilation workers | Isolated CPU containers | Parsing, XML, media inspection and packaging. |
| Workflow orchestration | Step Functions with AWS Batch, or approved equivalent | Dependencies, bounded retries, checkpoints and review gates. |
| Operator application | Accessible web interface | Exception review, progress and authorized release. |
| Delivery adapter | Agreed NLS integration | Submit validated packages and record acknowledgement. |

[AWS documents GPU jobs in Batch](https://docs.aws.amazon.com/batch/latest/userguide/gpu-jobs.html) and [Step Functions integration with Batch](https://docs.aws.amazon.com/step-functions/latest/dg/connect-batch.html). These capabilities support the design, but do not establish regional availability, account quotas, approval or required capacity.

## Jobs should be repeatable and resumable

Use immutable inputs, explicit output versions and idempotent job keys. A retried job should not duplicate a submission or overwrite an accepted book. Record checkpoints after durable artifacts are written and verified. Use separate states for failed processing, unresolved content, ready for review, and ready for delivery.

Keep model workers warm over a batch where appropriate. Loading weights for every paragraph wastes time, but the right worker lifetime depends on measured throughput and utilization.

## Capacity without invented benchmarks

A useful planning equation is:

```text
GPU-hours of work = source audio-hours × processing factor × retry multiplier
Wall-clock hours ≈ GPU-hours of work ÷ (GPU count × effective utilization)
```

Here the processing factor must be measured in GPU-hours per source audio-hour for the chosen workload. If it is a combined factor, avoid counting transcription or repair twice.

For illustration only: 30,000 hours × 0.5 GPU-hours per hour = 15,000 GPU-hours before retries. At a 1.2 retry multiplier, this becomes 18,000. Ten GPUs at 75% effective utilization would imply about 2,400 elapsed hours of compute processing. These are assumptions and arithmetic, not observed performance or a calendar delivery forecast. Human review, CPU work, queues and delivery can remain bottlenecks.

Benchmark the two input paths separately. Commercial recordings may need full-book transcription/alignment but only supplemental TTS; electronic text needs full narration. Measure accepted output, including failed attempts, rather than quoting isolated generation speed.

## Security and source handling

The draft references federal controls in **Section H**, which is absent from the supplied attachments. It also calls for source/output protection, retention/destruction procedures and security documentation. Obtain that missing material before fixing the controls baseline. [SOW §§3.6, 3.13.1, 5.1](https://sam.gov/api/prod/opps/v3/opportunities/resources/files/34eacfe92b1a46228b9f90674c4c60e1/download).

Proposed defaults include isolated inference in the approved environment, restricted unnecessary egress, pinned dependencies, least-privilege roles, separate content-protection keys, and limited source text in logs. Book content must be treated as data, including text that looks like instructions. SOW §5.1 limits use of Government source materials to contract performance and requires secure disposal at contract end. Implement those obligations alongside the agreed retention and deletion procedures.

Storage encryption, transfer encryption and PDTB protection solve different problems. Keep their responsibilities and key handling explicit.

## Handover and maintenance

Deliver source code, deployment instructions, configuration, dependency/model/voice inventory, operator and administrator guides, recovery procedures, validation tools and a known reference build. Confirm which third-party components can be redistributed and which must be supplied by NLS.

The draft specifies support response within 24 hours and a 12-month defect-correction warranty after acceptance; it permits ongoing support/maintenance separately from prohibited recurring access requirements. See [the milestone page](roadmap.md) and [SOW §§3.8–3.9, 3.12](https://sam.gov/api/prod/opps/v3/opportunities/resources/files/34eacfe92b1a46228b9f90674c4c60e1/download).

## Release evidence

For each released software/profile version, preserve migration notes, a reproducible deployment recipe, dependency hashes, regression results and the reference-book outputs. That makes later maintenance possible without relying on the original contractor's environment.