Usage

This page covers the main module workflows, from basic setup to additional storage and bucket features.

Quick start

The following steps enable the module, create object storage, and provide access to a bucket.

Enabling the module

d8 k patch moduleconfig deckhouse --type=merge --patch '{"spec":{"settings":{"allowExperimentalModules":true}}}'

Enable the module with a ModuleConfig:

d8 k apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: sds-object
spec:
  enabled: true
  version: 1
EOF

Creating storage

Storage is the data plane. Each supported backend uses its own Kind.

An example of creating a SeaweedFS storage class on top of an existing StorageClass using the SeaweedFSStore backend:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: default
spec:
  masters: 3
  volumeServers: 3
  replication: "001"        # One extra copy on another volume server.
  storage:
    sizePerNode: 50Gi
    class: localpath
EOF

The component counts and the replication code are SeaweedFS’s own settings and must agree. The third digit of the replication code sets the number of extra copies on other volume servers, so volumeServers must be at least that value plus one. Otherwise, SeaweedFS cannot complete writes.

To use several filers, set the filers and metadataStore fields in the SeaweedFSStore’s spec: the default value metadataStore takes, LevelDB, keeps metadata on each filer’s own volume and does not support sharing it across filers.

spec:
  filers: 3
  metadataStore: Postgres   # Requires the managed-postgres module.

Creating an object storage class

ObjectStore defines a storage class available to users. It references the storage and sets default parameters for the buckets it creates.

An example of an ObjectStore referencing the SeaweedFSStore created above:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ObjectStore
metadata:
  name: standard
spec:
  storeRef:
    kind: SeaweedFSStore
    name: default
  reclaimPolicy: Retain
  quota:
    maxSize: 100Gi          # Maximum bucket size; larger requests are rejected.
EOF

Check the storage status:

d8 k get seaweedfsstore

Example output:

NAME      VOLUMES   REPLICATION   PHASE   ENDPOINT                                    READY   AGE
default   3         001           Ready   http://default-seaweedfs.d8-sds-object...   True    3m

Once the data plane is deployed, the storage becomes Ready, and its endpoint appears in the ENDPOINT column.

Check the ObjectStore status:

d8 k get objectstore

Example output:

NAME       STORE-KIND       STORE     RECLAIM   PHASE   READY   AGE
standard   SeaweedFSStore   default   Retain    Ready   True    2m

An ObjectStore becomes Ready once the storage it references exists and is itself Ready.

The admission webhook rejects an unknown storeRef.kind value. The supported storage Kinds come from the controller’s driver registry, not from the CRD schema, so adding a new backend does not require changing the ObjectStore schema.

Requesting a bucket

Create the application namespace if it does not exist:

d8 k create namespace my-app

To create an S3 bucket, create a Bucket. The controller creates a cluster-wide BucketContents object for it, resolves the storage through the ObjectStore, and creates the bucket in the selected backend. BucketContents is linked to the originating Bucket and keeps track of where the data lives.

An example of creating a Bucket with accessPolicy: Private and reclaimPolicy: Retain:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  name: app-data
  namespace: my-app
spec:
  objectStoreRef: standard
  accessPolicy: Private
  reclaimPolicy: Retain     # Do not set this if you want to inherit the class value.
EOF

Check the Bucket status:

d8 k -n my-app get bucket app-data

Example output:

NAME       CONTENTS              PHASE   READY   AGE
app-data   contents-1f2e3d-...   Ready   True    20s

In the Ready state, the Bucket shows the name of its BucketContents.

The created BucketContents is visible cluster-wide (d8 k get bktc), but it should not be created by hand. The admission webhook allows creating this resource only from the module’s own ServiceAccount. This prevents a BucketContents from appearing without a matching user-created Bucket.

If the bucket already exists

The bucket name in the backend is derived from BucketContents, so a bucket with that name may already exist for reasons unrelated to the module: it was created by hand, it survived a storage recreation, or it belongs to another cluster using the same backend.

The module tags every bucket it creates with two tags:

storage.deckhouse.io/owned-by          sds-object
storage.deckhouse.io/bucket-contents   contents-1a2b3c4d5e-team-a-data

If these tags are missing, the module does not take the existing bucket under management. BucketContents stays not ready, and the BucketReady condition gets the reason BucketNotOwnedByModule. The existing bucket’s contents are left untouched.

Example BucketReady condition state:

BucketReady   False   BucketNotOwnedByModule
  bucket "team-a-data" already exists in the backend and is not managed by this
  module (it carries no ownership tag); remove it or point this Bucket at
  another store

In this situation, either delete the conflicting bucket if it is no longer needed, or point the Bucket at different storage. The controller does not retry automatically, so it never grants access to data it does not manage.

Buckets the module created before these tags existed are handled separately. On the next reconcile, the module identifies them by the stored status.bucketName and adds the missing tags. An upgrade therefore does not turn existing buckets into failures.

Requesting credentials

To get credentials, create a BucketAccess that references a Bucket in the same namespace with the Bound condition set to True. The controller generates a separate access key and secret key for this bucket and stores them in a Secret named <ACCESS>-s3-credentials by default.

An example of creating a BucketAccess referencing the Bucket app-data with permission: ReadWrite:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: BucketAccess
metadata:
  name: app-data
  namespace: my-app
spec:
  bucketRef: app-data
  permission: ReadWrite   # Use ReadOnly for read-only access.
EOF

Check the BucketAccess status:

d8 k -n my-app get bucketaccess app-data

Example output:

NAME       BUCKET     PHASE   SECRET                    READY   AGE
app-data   app-data   Ready   app-data-s3-credentials   True    20s

Once the credentials are created, BucketAccess shows the name of the Secret they are stored in.

Using the credentials

The Secret contains the standard S3 connection variables. Add it to the container in the Deployment manifest through envFrom:

Example Deployment fragment with the Secret attached:

spec:
  template:
    spec:
      containers:
        - name: app
          envFrom:
            - secretRef:
                name: app-data-s3-credentials

Storage on Ceph RGW

Storage on Ceph RGW is created through an SDSElasticStore object. It deploys a Ceph RADOS Gateway on top of an existing sds-elastic cluster and accepts Ceph’s own pool settings.

An example of creating an SDSElasticStore with size: 3 replication:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SDSElasticStore
metadata:
  name: heavy
spec:
  elasticClusterRef: main
  dataPool:
    replicated:
      size: 3
    # Alternatively, use erasure coding instead of replication.
    # erasureCoded: { dataChunks: 4, codingChunks: 2 }
EOF

An example of creating an ObjectStore referencing the SDSElasticStore:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ObjectStore
metadata:
  name: capacity
spec:
  storeRef:
    kind: SDSElasticStore
    name: heavy
EOF

Rotating credentials

To change the access key for a BucketAccess, set or change the storage.deckhouse.io/rotate annotation. The controller issues a new key pair, updates the Secret, and revokes the previous key:

d8 k -n my-app annotate bucketaccess app-data storage.deckhouse.io/rotate="$(date +%s)" --overwrite

Using an external metadata database

metadataStore: External lets you use an external PostgreSQL instance that the module does not deploy or operate. As with metadataStore: Postgres, several filers can share one metadata database. The administrator is responsible for the external database’s availability, backups, and upgrades.

If the metadata database is unavailable, the storage cannot serve S3 requests. Data replication across volume servers does not substitute for the metadata database’s own availability.

An example of creating a Secret and a SeaweedFSStore that uses it:

d8 k apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: media-metadata-db
  namespace: d8-sds-object
stringData:
  host: pg.example.internal
  port: "5432"
  database: seaweedfs_media
  username: seaweedfs
  password: <PASSWORD>
  sslmode: verify-full
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
---
apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: media
spec:
  metadataStore: External
  externalMetadataStore:
    secretRef:
      name: media-metadata-db
  filers: 3
  storage:
    class: linstor-r2
EOF

Database requirements. Use PostgreSQL 12 or newer, a separate empty database, and a role allowed to create tables in it. The filer creates one table per bucket on first use and does not run schema migrations.

Do not use a shared application database: table names are derived from bucket names and can collide with table names used by other applications.

TLS. The default is sslmode: require: the connection is encrypted, but the server certificate is not verified. To also verify the certificate and server name, add the CA certificate to the Secret’s ca.crt key and set sslmode: verify-full. The CA certificate is mounted into the filer and used when establishing the connection.

sslmode: disable is rejected during configuration validation, since the connection carries the password and object metadata. A typo or another sslmode value libpq does not recognize can pass the database reachability check — that check only verifies the DNS/TCP connection to the given address and port. The error surfaces when the filer starts: libpq rejects the value, and the filer pod keeps restarting.

Changing connection parameters. After you change the database address, password, or CA certificate in the Secret, the filers restart and load the new parameters. Connection parameters are read only when the process starts.

With a single filer, the S3 endpoint is briefly unavailable. With several filers, they restart one at a time.

Pod placement

The spec.placement parameter sets nodeSelector and tolerations. These settings apply to all SeaweedFSStore pods the module creates.

Master and volume-server replicas require placement on separate nodes. If no suitable separate node is available for a replica, the corresponding pod stays in the Pending state.

For filers, spreading across separate nodes is preferred. If that placement is not possible, several filers can run on the same node. Losing several filers reduces service availability until the pods are rescheduled, but does not cause metadata loss, since metadata is stored separately.

The spec.placement parameter does not apply to the managed metadata database. PostgreSQL placement is configured separately in a PostgresClass, using tolerations, nodeSelector, and nodeAffinity. To use a specific PostgresClass, set it in spec.postgresClassName. If the parameter is not set, the default class is used.

Example storage placement configuration with a dedicated PostgresClass for the metadata database:

spec:
  metadataStore: Postgres
  postgresClassName: storage-dedicated
  placement:
    nodeSelector:
      node-role/storage: ""
    tolerations:
      - key: storage.deckhouse.io/dedicated
        operator: Exists
        effect: NoSchedule

Responsibility for the external database. The module does not create the database or role, does not run migrations, does not create backups, and does not monitor the PostgreSQL server’s health. Before configuring the filer, the module checks that the given address and port are reachable. A DNS resolution error or an unreachable port is reflected in the storage status.

Example status message:

BackendReady   False   Pending
  the external metadata database at pg.example.internal:5432 did not answer:
  dial tcp: lookup pg.example.internal: no such host

If the metadata database is lost, objects remain on the volume servers, but the mapping between objects and buckets is lost. Include this database in the storage’s backup plan.

Reclaim policy

What happens to stored data on deletion depends on the reclaim policy (reclaimPolicy parameter):

• For a Bucket with reclaimPolicy: Retain, corresponding bucket and its objects are kept. BucketContents moves to the Released phase and keeps a record of the original Bucket and the storage where the data lives:

  d8 k get bktc

  Example output:

  NAME                  STORE-KIND       STORE     OWNER-NS   OWNER      BUCKET            PHASE      READY   AGE
  contents-1f2e3d-...   SeaweedFSStore   default   my-app     app-data   my-app-app-data   Released   False   4h
  

  If you create a Bucket named app-data in the my-app namespace again, the controller links it back to the retained BucketContents and its data. The same happens when the namespace is deleted: the Bucket is removed with it, while Retain data survives.

  Delete instead removes the bucket with its objects, and BucketContents along with them. A Bucket that does not set the policy inherits it from its class; the class default is Retain.

• Deleting a BucketAccess always revokes its access key and deletes its Secret (the bucket’s data is not affected).

• Storage reclaimPolicy: Retain (the default) — deleting the storage keeps the data: SDSElasticStore keeps its Ceph RGW pools, SeaweedFSStore keeps its PVCs. Delete destroys the data.

• Deleting an ObjectStore does not delete the storage or any buckets already created through it. After the class is deleted, you can no longer create Buckets through it. The reference to the actual storage is kept in BucketContents.

To delete data retained after a Bucket with reclaimPolicy: Retain was deleted, delete the corresponding BucketContents. If the linked Bucket is already gone, the controller deletes the bucket in the backend.

The reclaim policy governs what happens when a Bucket is deleted. Deleting BucketContents itself is a separate operation. If the Bucket is already gone, deleting BucketContents deletes the retained bucket in the backend. If the linked Bucket still exists, the controller recreates BucketContents and links it back to the existing bucket.

Publishing storage outside the cluster

By default, storage is reachable only from inside the cluster: status.endpoint has only the internal field filled in, and that address is what goes into the credentials Secret. To make storage usable from outside, publish it through the alb module, which implements the Gateway API on top of Envoy.

Before publishing, prepare the following:

1. Create a Gateway using an ALBInstance or ClusterALBInstance from the alb module. Get the Gateway’s name and namespace from that object’s status.
2. Create a DNS record for the external endpoint’s hostname.
3. Create a kubernetes.io/tls Secret with a certificate for that hostname in the d8-sds-object namespace. The module does not issue certificates, so use a Secret created by cert-manager or one you have prepared in advance.

Then add a publish block to the storage:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: default
spec:
  storage:
    class: linstor-thin-r2
  publish:
    hostname: s3.example.com
    gatewayRef:
      name: public-gw      # The Gateway object created by the administrator.
      namespace: d8-alb
    tls:
      secretRef:
        name: s3-example-com-tls
EOF

The module creates a ListenerSet (host, port 443, TLS termination with the given certificate) and an HTTPRoute to the storage’s S3 port. Check the published endpoint:

d8 k get seaweedfsstore default -o jsonpath='{.status.endpoint}'

Example output:

{"external":"https://s3.example.com","internal":"http://default-seaweedfs...:8333","region":"us-east-1"}

TLS is mandatory and cannot be disabled: S3 credentials travel in the Authorization header, so publishing over plain HTTP would let anyone watching the traffic intercept them.

The module publishes the hostname, and the listener uses port 443. If the ALBInstance or ClusterALBInstance is configured with a HostPort inlet on a non-standard port, for example 8443, status.endpoint.external contains https://<HOST> without a port number. In that environment, the client must explicitly use https://<HOST>:8443.

For a LoadBalancer inlet, which accepts connections on port 443, the address from status.endpoint.external can be used as is.

Choosing the address in the credentials

By default, the credentials Secret gets the storage’s in-cluster address. To use the external address, set endpointScope: External on the BucketAccess.

An example of creating a BucketAccess with endpointScope: External:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: BucketAccess
metadata:
  name: backup-writer
  namespace: my-app
spec:
  bucketRef: app-data
  permission: ReadWrite
  endpointScope: External
EOF

If a BucketAccess requests the external address but the storage is not published, BucketAccess does not fall back to the in-cluster address automatically. The object stays not ready, and its condition reports that spec.publish needs to be configured on the storage.

When the endpoint changes (publishing was enabled, or the hostname changed), the Secrets of all affected BucketAccess objects are reissued automatically.

Addressing limits

Path-style addressing is supported: https://s3.example.com/<BUCKET>/<KEY>. It needs only one DNS name and one certificate.

Virtual-hosted addressing (https://<BUCKET>.s3.example.com/<KEY>) is represented in the API as spec.publish.addressing: VirtualHosted, but is currently rejected by the admission webhook. It requires a wildcard DNS name and a wildcard certificate, and support for it varies across the backends’ S3 gateways and has not been verified by the module yet. As a result, a published endpoint supports path-style addressing only.

Components available only inside the cluster

For SeaweedFS, only the S3 port is published externally. The module creates a separate Service with just that port for it. The storage’s main Service also carries the filer’s HTTP and gRPC APIs, so it is not used for external publishing.

In Ceph RGW, the S3 API and the Admin Ops API share a port, and administrative methods are available under /admin/… paths. As a result, publishing an SDSElasticStore also makes that path reachable from the network.

Credentials the module issues to users carry no administrative rights and do not grant access to the Admin Ops API. However, Rook creates a user named rgw-admin-ops-user with buckets=*;users=* capabilities for every object store. Its keys are stored in a Secret in the d8-sds-elastic namespace. The security of the Admin Ops API depends on protecting these credentials and on restrictions at the Gateway level.

Before publishing Ceph RGW storage, consider the following protections:

• Deny access to /admin on the Gateway, if your configuration supports it.
• If restricting /admin is not possible, consider a separate CephObjectStore for the external endpoint.
• Treat the rgw-admin-ops-user Secret as administrative credentials exposed to the internet.

SeaweedFS has no such exposure: the external Service carries only the S3 port, and the filer’s own APIs stay unreachable through the external route.

Diagnosing external publishing

The state of external publishing is reflected in a separate PublishedEndpointReady condition and does not affect the storage’s own Ready state. A broken external route does not block creating or deleting buckets, or issuing credentials for in-cluster access.

Check the PublishedEndpointReady condition:

d8 k get seaweedfsstore default -o jsonpath='{range .status.conditions[?(@.type=="PublishedEndpointReady")]}{.reason}: {.message}{end}'

Common reasons:

Public read

accessPolicy: PublicRead on a Bucket opens its objects for anonymous reading. Both backends support this mode.

An example of creating a Bucket with accessPolicy: PublicRead:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  name: site-assets
  namespace: my-app
spec:
  objectStoreRef: standard
  accessPolicy: PublicRead
EOF

Verify that the object can be read without credentials:

curl -s https://s3.example.com/my-app-site-assets/logo.png -o logo.png

Public access grants only reading objects:

• Objects only, no listing. An anonymous GetObject call succeeds if the object key is known. ListObjects for the bucket stays forbidden. Public read access does not include a public listing of the bucket’s contents.
• Read only. Anonymous write, delete, and tagging are forbidden on both backends.
• This bucket only. The grant applies to this bucket’s name; it does not affect other buckets in the same storage.

Keep these two things in mind when using PublicRead:

• Public access depends on the endpoint being reachable. Inside the cluster, the S3 endpoint is reachable through the module’s Service. For access from outside, configure spec.publish on the storage. Publishing makes the endpoint reachable over the network; accessPolicy decides whether reading a given bucket requires authorization.
• A Bucket in the Released phase stays public. Deleting a Bucket with reclaimPolicy: Retain keeps the data and the BucketContents itself, along with its accessPolicy. As a result, the data stays available in the same mode after the Bucket is deleted. To turn off public read, an administrator with permission to update cluster-scoped BucketContents must set accessPolicy: Private on the BucketContents. If the retained data is no longer needed, the administrator can delete BucketContents instead. The module’s built-in User and ClusterEditor roles do not grant write access to BucketContents.

After changing the value to Private, public read access is turned off at the next reconcile. Access keys already issued for the bucket are not affected.

Versioning and object lock

versioning: Enabled keeps every version of an object instead of overwriting it. objectLock turns on WORM (Write Once Read Many) behavior for object versions. A version under an active retention period cannot be deleted before it ends — not by the user, not by the storage administrator, and not by the module itself.

An example of creating a Bucket with versioning and Object Lock enabled:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  name: audit-log
  namespace: my-app
spec:
  objectStoreRef: standard
  versioning: Enabled
  objectLock:
    mode: Compliance     # Allowed values: Governance or Compliance.
    days: 365
  reclaimPolicy: Retain  # Required together with objectLock.
EOF

Governance mode allows bypassing the restriction with the right permission. In Compliance mode, no one can delete a protected version before its retention period ends. The retention period can be extended, but not shortened. The backend enforces these rules.

The module reflects the actual state of these settings in the backend, in status:

d8 k get bktc -o custom-columns='NAME:.metadata.name,VERSIONING:.status.versioning,LOCK:.status.objectLock.enabled,MODE:.status.objectLock.mode'

API checks

The admission webhook rejects several combinations before they ever reach the backend:

• objectLock requires versioning: Enabled. Object lock is built on versions.
• objectLock is immutable and can only be set when the Bucket is created. Ceph RGW accepts Object Lock configuration only when the bucket is created, so it can never be added to an existing bucket. This restriction is the same on both backends: to protect existing data, create a new bucket with Object Lock and migrate the data.
• objectLock is incompatible with reclaimPolicy: Delete — including when Delete comes from the default class. A bucket with protected objects cannot be deleted, so that combination would leave BucketContents in Terminating until the last retention period expires. The request is rejected outright, not silently switched to Retain.
• Versioning cannot be turned off on a locked bucket. Both backends refuse this, and so does the API.

Legal hold

Legal hold blocks deleting an object version indefinitely and stays in effect until explicitly released. It is not configured on the Bucket: the client sets and clears legal hold for a specific object version using its own credentials:

mc legalhold set myalias/my-app-audit-log/report.pdf
mc legalhold clear myalias/my-app-audit-log/report.pdf

While legal hold is set, the protected version cannot be deleted by any means. To delete it, clear the legal hold first.

Deleting protected data

Deleting a Bucket with reclaimPolicy: Retain keeps the data, moves BucketContents to the Released phase, and keeps the objects’ protection settings in effect. To delete the retained data, delete the BucketContents. If the bucket still has protected versions, the backend refuses the deletion, BucketContents keeps its finalizer, and the reason appears in status.conditions.

Check the Ready condition message:

d8 k get bktc contents-... -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'

Example output:

the backend refuses to delete the bucket while its objects are protected; ...

Once the last retention period expires, deletion proceeds automatically. Before that, protected versions can only be deleted with permission to bypass retention. Compliance mode allows no such bypass.

Storage capacity

Storage capacity data is published in status.capacity and shown in the wide output of d8 k.

Check the storage capacity:

d8 k get seaweedfsstore media -o wide

Example output:

NAME    VOLUMES   REPLICATION   PHASE   ENDPOINT                     USED%   CAPACITY   READY
media   3         001           Ready   http://...svc:8333           37.24   300Gi      True

status.capacity fragment from the SeaweedFSStore YAML representation:

status:
  capacity:
    total: 300Gi
    used: 111Gi
    available: 189Gi
    usedPercent: "37.24"
    lastUpdated: "2026-08-30T09:12:04Z"

status.capacity is calculated differently for each backend:

• SeaweedFSStore — the sum of disk capacity used by this storage’s volume servers. If a volume server is unreachable while capacity is being collected, its capacity is excluded from the total. This keeps an unreachable disk from being counted as free space.
• SDSElasticStore — the raw capacity of the entire Ceph cluster, as reported by Ceph itself. This value is not a quota for this particular SDSElasticStore and does not show how much space an application can actually use: the replication scheme and usage by the cluster’s other pools both affect it.

If capacity data could not be obtained from either source, status.capacity is not populated. A missing field means the capacity is unknown; a zero value is not used for this case.

Metrics and alerts

Each storage exports the following metrics with the store_kind and store labels:

Capacity metrics are exported only after capacity data has been obtained successfully. If the data could not be obtained, the sds_object_store_capacity_bytes_* metrics are absent, so filling-up alerts do not fire based on an unknown value.

The sds_object_store_ready metric is exported at all times, as long as the corresponding storage exists. A value of 0 means the storage is not ready. The time series disappears once the resource or the module is deleted. This way, the not-ready state is distinguished from the resource not existing at all.

The following alerts are configured for these metrics: D8SdsObjectStoreNotReady — the storage stays not ready for more than 30 minutes; D8SdsObjectStoreFillingUp — over 85% of capacity used for an hour; D8SdsObjectStoreAlmostFull — over 95% of capacity used. Storage state is also shown on the SDS Object — Stores dashboard.

The 95% threshold accounts for SeaweedFS behavior: the master stops placing new volumes on a disk once it reaches 90% full. As a result, storage can stay Ready while no longer able to place new data.

Data integrity

Storage status carries details of the last integrity check: its source, when it ran, and any damage found.

Check the result of the last integrity check:

d8 k get seaweedfsstore,sdselasticstore -o custom-columns='NAME:.metadata.name,INTEGRITY:.status.conditions[?(@.type=="IntegrityHealthy")].status,LAST SCRUB:.status.integrity.lastScrubTime,DAMAGED:.status.integrity.damaged'

Before the first check completes, the IntegrityHealthy condition is Unknown. This distinguishes the absence of check results from a confirmed damage-free state. When a problem is found, the condition becomes False, and its message carries the backend’s own diagnostic details, including which disk to check.

The Ready state does not depend on IntegrityHealthy. Storage can keep serving the data that is still available even after a damaged volume is found.

Check mechanism and frequency

Each backend checks integrity differently:

• SeaweedFS — SeaweedFS verifies a checksum whenever an object is read. For data that applications have not accessed in a long time, the module schedules a full integrity check (scrub): SeaweedFS reads the stored data and verifies its checksums.

  Configure the periodic check in spec.integrity:

  apiVersion: storage.deckhouse.io/v1alpha1
  kind: SeaweedFSStore
  metadata:
    name: media
  spec:
    integrity:
      interval: 24h     # The default is 168h; values below 1h are raised to 1h.
      mode: Full        # Full is the default and reads every byte; Index checks only indexes.
      enabled: true
    # The rest of spec follows.

  Full is the default because Index cannot detect corruption in the object content. The default interval accounts for the load: a full check reads every stored byte.

  The full check runs asynchronously, outside of reconcile. While it runs, IntegrityHealthy carries the reason ScrubInProgress. The result is written to status when the check completes; on large storage, it can take a long time. Restarting the controller stops an in-progress check; the next one starts on schedule.

  An unreachable volume server is not treated as damaged. It appears in status.integrity.unreachable. If some nodes could not be checked and no damage was found, IntegrityHealthy stays Unknown with the reason ScrubIncomplete, since the check did not cover the whole storage.

  If storage has no volumes yet, a pass over it does not count as a completed data check. The condition stays Unknown with the reason NothingToCheck, and status.integrity is not populated. Once data appears, the storage is checked on schedule.

• Ceph RGW — Ceph performs checks on its own schedule. The module gets results from ElasticCluster health checks (OSD_SCRUB_ERRORS, PG_DAMAGED, PG_NOT_DEEP_SCRUBBED, …). SDSElasticStore has no spec.integrity: running deep-scrub and automatic repair are configured at the Ceph cluster level and are not managed by the sds-object module.

status.integrity.source shows the source of the published check result.

Replication status

A missing copy is not damage, so it is reported separately in the RedundancyHealthy condition and status.redundancy.

Check the replication state:

d8 k get seaweedfsstore -o custom-columns='NAME:.metadata.name,WANTED:.status.redundancy.copiesWanted,VOLUMES:.status.redundancy.volumes,SHORT:.status.redundancy.underReplicated'

Copy counts come from the SeaweedFS master’s own topology and are updated on every reconcile, independent of the integrity-check schedule.

RedundancyHealthy: False combined with IntegrityHealthy: True means no damage was found, but the copy count is below what is required. With replication: "000", only one copy is required, so that storage is never considered under-replicated as long as that one copy exists.

SeaweedFS 4.39 does not restore a lost replica automatically. Its maintenance framework includes balance, vacuum, erasure coding, EC balance, S3 lifecycle, and Iceberg tasks, but no dedicated replication-repair task, even though the protocol has a task type for it. So after losing a replica, restore it by hand. First, list the pods in the module namespace and choose a pod running the SeaweedFS image:

d8 k -n d8-sds-object get pods -o custom-columns='NAME:.metadata.name,IMAGES:.spec.containers[*].image'

Open a shell in the selected pod:

d8 k -n d8-sds-object exec -it <SEAWEEDFS_POD> -- sh

Then run the replication repair from that shell:

weed shell -master=<STORE>-seaweedfs-master:9333 <<'EOF'
lock
volume.fix.replication -apply
unlock
EOF

For SDSElasticStore, the condition is Unknown with the reason NotAccountedHere. Replication status is controlled by Ceph itself and available through its own health checks; the sds-object module does not duplicate that accounting.

Automatic repair

By default, the module only reports damage it finds. If you set spec.integrity.autoRepair: true, the module attempts to repair a damaged SeaweedFS copy from another copy that passed its check:

spec:
  replication: "001"      # Repair requires more than one copy.
  integrity:
    autoRepair: true

Automatic repair is off by default, because the procedure deletes the damaged copy and creates a new one. Before repairing each volume, the module checks the following conditions:

• More than one copy of the volume exists. With replication: "000", the damaged copy is the only one, so automatic repair is not possible.
• Another copy of the same volume passed the integrity check in the current cycle. An unreachable or unchecked copy is not used as a repair source.

If either condition is not met, the module does not delete the damaged copy, and status.integrity.details states why repair was skipped. If creating a new copy after removing the damaged one fails, RedundancyHealthy reports the insufficient copy count.

Successfully repaired volumes are counted in status.integrity.repaired and no longer count as damaged. Once repair succeeds, the corresponding alert stops firing.

None of this applies to Ceph RGW: Ceph itself manages placement group repair. The module does not run deep-scrub or enable osd_scrub_auto_repair.

Metrics and alerts

Integrity and replication status are also exported as metrics with the store_kind and store labels:

The following alerts are configured for these metrics:

• D8SdsObjectStoreDataDamaged — the last completed check found damage. Fires with no additional delay.
• D8SdsObjectStoreUnderReplicated — the copy count has been below what is required for 15 minutes. The delay rules out brief flaps during a planned volume-server restart.
• D8SdsObjectStoreEncryptionKeyWithheld — the encryption key value changed, but the module is not applying the new key, so previously written data does not become unreadable after the gateway restarts.
• D8SdsObjectStoreNotScrubbed — more than three spec.integrity.interval periods have passed since the last completed check. This catches the case where the periodic check has stopped running even though the last saved IntegrityHealthy result is still successful. If the periodic check is disabled, the interval metric is not published and this alert does not apply.

The Grafana dashboard SDS Object — Data integrity, in the Storage folder, shows the number of storage resources with damage, insufficient replication, and overdue checks, along with each storage’s state and time since its last integrity check.

Encryption at rest

Without spec.encryption, object content is stored on the volume servers without server-side encryption. Anyone who gets access to the disk can read that data. Encryption can be enabled without any changes to the applications writing the objects.

Create a Secret with the encryption key material:

# Generate 32 bytes of key material. Keep a backup copy of the key outside the cluster.
d8 k -n d8-sds-object create secret generic media-sse --from-literal=kek="$(head -c 32 /dev/urandom | xxd -p -c 32)"

An example of creating a SeaweedFSStore with server-side encryption enabled and the Secret referenced:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: media
spec:
  storage:
    class: linstor-thin-r2
  encryption:
    mode: ServerManaged
    keySecretRef:
      name: media-sse
EOF

Instead of kek, the Secret can hold a key field with a passphrase the module derives the key from. This is useful when the key material comes from a system that hands out strings rather than raw bytes.

Once encryption is enabled, the module gives the S3 gateway the wrapping key and turns on server-side encryption for its buckets. Each object is encrypted with its own data key, which is protected by the key from the Secret. The wrapping key is never stored in the filer’s metadata store, so access to the Postgres database or the filer’s PVC alone does not let anyone decrypt the objects’ contents.

Check the encryption state:

d8 k get swfsstore media -o jsonpath='{.status.encryption}' | jq

Example output:

{
  "mode": "ServerManaged",
  "keyFingerprint": "sha256:9c1f4a0b7e2d8536",
  "since": "2026-08-25T09:12:04Z"
}

Changing the encryption key

Objects already written are not re-wrapped automatically. If the gateway starts with a different wrapping key, reading objects encrypted under the previous key fails with an internal server error. To prevent losing access to data, the module compares the key fingerprint and refuses to apply a changed key without explicit confirmation.

Check the EncryptionActive condition message:

d8 k get swfsstore media -o jsonpath='{.status.conditions[?(@.type=="EncryptionActive")]}' | jq -r .message

Example output:

the wrapping key changed (sha256:9c1f4a0b7e2d8536 -> sha256:22ba07f6c4e1d980) and was NOT applied: ...

While the condition is False, the gateway keeps using the previous key, under which previously written data stays readable. To return to a normal state, restore the original key material. If the data encrypted under the previous key is genuinely no longer needed, you can confirm the new key instead:

d8 k annotate swfsstore media storage.deckhouse.io/encryption-key-change-acknowledged=sha256:22ba07f6c4e1d980

The gateway reads the key at startup. So changing the Secret by itself does not affect an already running pod, but without this safeguard, the new key would be applied at the next restart, and previously encrypted objects could become unreadable.

Once spec.encryption.mode is enabled, server-side encryption cannot be turned off for existing storage. Previously encrypted objects still need the original key. To move to storage without server-side encryption, create new storage and migrate the data to it.

Encryption limitations and SSE-C

Server-side encryption has the following limitations and can be combined with client-side encryption:

• Object names, sizes, and tags are not encrypted — only the content is.
• Objects written before encryption was enabled are not encrypted automatically. To encrypt them, upload the data again after enabling encryption.
• A client can use its own key (SSE-C). The key is sent with every request and is not stored by the storage. SSE-C does not depend on spec.encryption. Use SSE-C only through a published TLS endpoint: on the internal HTTP endpoint, the key travels in the header without transport encryption.
• SSE-KMS is not supported for SeaweedFSStore. The gateway reads KMS configuration only from a static credentials file, which is incompatible with the keys the module creates for each BucketAccess. For external KMS, use SDSElasticStore (Ceph RGW).

Encryption in Ceph RGW

For SDSElasticStore, server-side encryption requires an external key management system (KMS). Ceph RGW gets keys from the KMS and does not support handing a key directly to the RGW process. Deckhouse Stronghold with its transit engine can be used as the KMS.

Create a Secret with the Stronghold token:

# Token that may only read and use the transit key.
d8 k -n d8-sds-object create secret generic rgw-stronghold-token --from-literal=token="$STRONGHOLD_TOKEN"

An example of creating an SDSElasticStore with the Secret and Stronghold address referenced:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SDSElasticStore
metadata:
  name: heavy
spec:
  elasticClusterRef: main
  encryption:
    mode: ServerManaged
    stronghold:
      address: https://stronghold.d8-stronghold.svc.cluster.local:8200
      tokenSecretRef:
        name: rgw-stronghold-token
      # For Stronghold with a private certificate, configure a CA Secret.
      # caSecretRef:
      #   name: stronghold-ca
EOF

Keep the following in mind when configuring KMS for Ceph RGW:

• Stronghold is used through the Vault API. Rook gets KMS_PROVIDER: vault and VAULT_ADDR, because Stronghold implements the Vault API that Ceph RGW speaks. Any other Vault-compatible store can work the same way.
• The transit mount path is not configurable. For SSE-S3, Rook builds the RGW prefix from a single engine name (/v1/transit), so a transit engine mounted elsewhere would be accepted here and never used. Mount it as transit.
• The token is copied to the d8-sds-elastic namespace. Rook expects a Secret with the token next to the CephObjectStore, not next to the storage object, so the module keeps a managed copy there. Deleting the storage also deletes the copy. After the source token is rotated, the copy is updated on the next reconcile. The CA bundle is copied the same way: the ca.crt field is converted to the cert field Rook expects.
• No key fingerprint is kept in status.encryption. The key itself lives in Stronghold and is never handed to the module, so the module cannot compute its fingerprint or detect a key change. The availability of previously encrypted data depends on the key surviving in Stronghold.

status.encryption.message and the EncryptionActive condition reflect errors applying the encryption configuration in RGW, most often a missing token Secret. For existing storage, the rest of reconcile continues. New storage with encryption enabled is not created until the KMS configuration is ready, so the first objects are never written unencrypted.

Automatic object deletion

To automatically remove outdated logs, exports, or build artifacts, use spec.lifecycle. Lifecycle rules are applied by the S3 backend itself.

An example of creating a Bucket with deletion rules in spec.lifecycle:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  namespace: team-a
  name: build-logs
spec:
  objectStoreRef: standard
  lifecycle:
    rules:
      - id: logs
        prefix: logs/
        expireAfterDays: 30
      - id: leftovers
        abortIncompleteUploadsAfterDays: 1
EOF

Every rule must have a unique id. This identifier is passed to the S3 backend and used to match rules when the configuration is updated. It is set explicitly and does not depend on the rule’s position in the list.

• expireAfterDays deletes an object this many days after it was written. On a versioned bucket, this only makes the current version noncurrent — the data stays, and expireNoncurrentAfterDays frees the space.
• expireNoncurrentAfterDays deletes a version this many days after it stopped being current.
• abortIncompleteUploadsAfterDays removes parts of multipart uploads that never completed, after the given number of days. These parts take up space but do not show up as regular objects.
• prefix limits the rule to objects whose keys start with the given prefix. If prefix is not set, the rule applies to every object in the bucket.

On every reconcile, the module builds the full set of lifecycle rules from spec and sends it to the backend. If a rule is removed from the Bucket, it is also removed from the bucket’s configuration in the backend.

Object Lock takes priority over lifecycle deletion. If a version has an active retention period, a lifecycle rule cannot delete it before that period ends.

Transitions between storage classes are not supported. The current lifecycle API only supports deletion operations. SeaweedFS 4.39 supports expiration by days and by date, deleting noncurrent versions, newer-noncurrent limits, aborting incomplete uploads, and cleaning up delete markers, but does not support the Transition action. Ceph RGW can move objects between storage classes, but a transition parameter common to both backends is not available yet.

S3 operation support

SeaweedFS and Ceph RGW support different sets of S3 operations. Before relying on an application feature, check that it is supported by the backend you chose.

The table is based on the S3 API implementation in SeaweedFS 4.39 and on the support matrix of the Ceph RGW version used by the sds-elastic module. These are the backends’ own capabilities, not an additional implementation by the sds-object module. The table needs to be re-checked whenever the backend versions are updated.

Partially implemented SeaweedFS operations. Some S3 endpoints respond successfully but do not persist any configuration. For example, the accelerate endpoint always returns Suspended, bucket logging returns an empty status, and the analytics, inventory, intelligent-tiering, and metrics listings all come back empty. A 200 response for these operations confirms the request was handled, not that the corresponding feature exists.

What the module manages, and what passes through. The module manages bucket creation, bucket policy for accessPolicy, quotas, versioning, Object Lock, and default encryption. Every other S3 operation runs directly between the application and the backend; the module neither configures nor tracks it.

If a requested feature is not supported by the chosen backend, BucketContents reflects that in the FeaturesApplied condition.

Example FeaturesApplied condition state:

FeaturesApplied   False   Unsupported
  backend SeaweedFS does not enforce: quota.maxObjects

The table above describes the backends’ general capabilities, while the FeaturesApplied condition shows the result of applying the requested settings to a specific bucket.

Diagnostics

Every object carries an overall state in status.phase, with the detailed reason for the current state in status.conditions:

• Pending — the controller has not processed the object yet, or is waiting on a dependent stage that has not reported anything yet.
• InProgress — processing is underway: the stage has started but has not yet confirmed success.
• Ready — every stage confirmed success. Bucket data, BucketAccess credentials, and the storage endpoint are current only in this phase.
• Error — one of the processing stages failed; the reason is in the failing condition’s message.
• Released (BucketContents only) — the owning Bucket was deleted with reclaimPolicy: Retain; the data is preserved and will be linked again to a Bucket with the same name in the same namespace.

To see the details, run:

d8 k get bucket app-data -n my-app -o jsonpath='{.status.conditions}'

Common messages, where they appear, and what to do: