Compose lets you define AI models as core components of your application, so you can declare model dependencies alongside services and run the application on any platform that supports the Compose Specification.
Prerequisites
- Docker Compose v2.38 or later
- A platform that supports Compose models such as Docker Model Runner (DMR) or compatible cloud providers. If you are using DMR, see the requirements.
What are Compose models?
Compose models are a standardized way to define AI model dependencies in your application. By using the
models top-level element in your Compose file, you can:
- Declare which AI models your application needs
- Specify model configurations and requirements
- Make your application portable across different platforms
- Let the platform handle model provisioning and lifecycle management
Basic model definition
To define models in your Compose application, use the models top-level element:
This example defines:
- A service called
chat-appthat uses a model namedllm - A model definition for
llmthat references theai/smollm2model image
Model configuration options
Models support various configuration options:
Common configuration options include:
model(required): The OCI artifact identifier for the model. This is what Compose pulls and runs via the model runner.context_size: Defines the maximum token context size for the model.NoteEach model has its own maximum context size. When increasing the context length, consider your hardware constraints. In general, try to keep context size as small as feasible for your specific needs.
runtime_flags: A list of raw command-line flags passed to the inference engine when the model is started. For example, if you use llama.cpp, you can pass any of the available parameters.Platform-specific options may also be available via extension attributes
x-*
TipSee more example in the Common runtime configurations section.
Service model binding
Services can reference models in two ways: short syntax and long syntax.
Short syntax
The short syntax is the simplest way to bind a model to a service:
With short syntax, the platform automatically generates environment variables based on the model name:
LLM_URL- URL to access the LLM modelLLM_MODEL- Model identifier for the LLM modelEMBEDDING_MODEL_URL- URL to access the embedding-modelEMBEDDING_MODEL_MODEL- Model identifier for the embedding-model
Long syntax
The long syntax allows you to customize environment variable names:
With this configuration, your service receives:
AI_MODEL_URLandAI_MODEL_NAMEfor the LLM modelEMBEDDING_URLandEMBEDDING_NAMEfor the embedding model
Platform portability
One of the key benefits of using Compose models is portability across different platforms that support the Compose specification.
Docker Model Runner
When Docker Model Runner is enabled:
Docker Model Runner will:
- Pull and run the specified model locally
- Provide endpoint URLs for accessing the model
- Inject environment variables into the service
Cloud providers
The same Compose file can run on cloud providers that support Compose models:
Cloud providers might:
- Use managed AI services instead of running models locally
- Apply cloud-specific optimizations and scaling
- Provide additional monitoring and logging capabilities
- Handle model versioning and updates automatically
Common runtime configurations
Below are some example configurations for various use cases.
Development
Conservative with disabled reasoning
Creative with high randomness
Highly deterministic
Concurrent processing
Rich vocabulary model
Alternative configuration with provider services
ImportantThis approach is deprecated. Use the
modelstop-level element instead.
You can also use the provider service type, which allows you to declare platform capabilities required by your application.
For AI models, you can use the model type to declare model dependencies.
To define a model provider:
