* feat(filer): add lazy directory listing for remote mounts
Directory listings on remote mounts previously only queried the local
filer store. With lazy mounts the listing was empty; with eager mounts
it went stale over time.
Add on-demand directory listing that fetches from remote and caches
results with a 5-minute TTL:
- Add `ListDirectory` to `RemoteStorageClient` interface (delimiter-based,
single-level listing, separate from recursive `Traverse`)
- Implement in S3, GCS, and Azure backends using each platform's
hierarchical listing API
- Add `maybeLazyListFromRemote` to filer: before each directory listing,
check if the directory is under a remote mount with an expired cache,
fetch from remote, persist entries to the local store, then let existing
listing logic run on the populated store
- Use singleflight to deduplicate concurrent requests for the same directory
- Skip local-only entries (no RemoteEntry) to avoid overwriting unsynced uploads
- Errors are logged and swallowed (availability over consistency)
* refactor: extract xattr key to constant xattrRemoteListingSyncedAt
* feat: make listing cache TTL configurable per mount via listing_cache_ttl_seconds
Add listing_cache_ttl_seconds field to RemoteStorageLocation protobuf.
When 0 (default), lazy directory listing is disabled for that mount.
When >0, enables on-demand directory listing with the specified TTL.
Expose as -listingCacheTTL flag on remote.mount command.
* refactor: address review feedback for lazy directory listing
- Add context.Context to ListDirectory interface and all implementations
- Capture startTime before remote call for accurate TTL tracking
- Simplify S3 ListDirectory using ListObjectsV2PagesWithContext
- Make maybeLazyListFromRemote return void (errors always swallowed)
- Remove redundant trailing-slash path manipulation in caller
- Update tests to match new signatures
* When an existing entry has Remote != nil, we should merge remote metadata into it rather than replacing it.
* fix(gcs): wrap ListDirectory iterator error with context
The raw iterator error was returned without bucket/path context,
making it harder to debug. Wrap it consistently with the S3 pattern.
* fix(s3): guard against nil pointer dereference in Traverse and ListDirectory
Some S3-compatible backends may return nil for LastModified, Size, or
ETag fields. Check for nil before dereferencing to prevent panics.
* fix(filer): remove blanket 2-minute timeout from lazy listing context
Individual SDK operations (S3, GCS, Azure) already have per-request
timeouts and retry policies. The blanket timeout could cut off large
directory listings mid-operation even though individual pages were
succeeding.
* fix(filer): preserve trace context in lazy listing with WithoutCancel
Use context.WithoutCancel(ctx) instead of context.Background() so
trace/span values from the incoming request are retained for
distributed tracing, while still decoupling cancellation.
* fix(filer): use Store.FindEntry for internal lookups, add Uid/Gid to files, fix updateDirectoryListingSyncedAt
- Use f.Store.FindEntry instead of f.FindEntry for staleness check and
child lookups to avoid unnecessary lazy-fetch overhead
- Set OS_UID/OS_GID on new file entries for consistency with directories
- In updateDirectoryListingSyncedAt, use Store.UpdateEntry for existing
directories instead of CreateEntry to avoid deleteChunksIfNotNew and
NotifyUpdateEvent side effects
* fix(filer): distinguish not-found from store errors in lazy listing
Previously, any error from Store.FindEntry was treated as "not found,"
which could cause entry recreation/overwrite on transient DB failures.
Now check for filer_pb.ErrNotFound explicitly and skip entries or
bail out on real store errors.
* refactor(filer): use errors.Is for ErrNotFound comparisons
* feat(remote): add -noSync flag to skip upfront metadata pull on mount
Made-with: Cursor
* refactor(remote): split mount setup from metadata sync
Extract ensureMountDirectory for create/validate; call pullMetadata
directly when sync is needed. Caller controls sync step for -noSync.
Made-with: Cursor
* fix(remote): validate mount root when -noSync so bad bucket/creds fail fast
When -noSync is used, perform a cheap remote check (ListBuckets and
verify bucket exists) instead of skipping all remote I/O. Invalid
buckets or credentials now fail at mount time.
Made-with: Cursor
* test(remote): add TestRemoteMountNoSync for -noSync mount and persisted mapping
Made-with: Cursor
* test(remote): assert no upfront metadata after -noSync mount
After remote.mount -noSync, run fs.ls on the mount dir and assert empty
listing so the test fails if pullMetadata was invoked eagerly.
Made-with: Cursor
* fix(remote): propagate non-ErrNotFound lookup errors in ensureMountDirectory
Return lookupErr immediately for any LookupDirectoryEntry failure that
is not filer_pb.ErrNotFound, so only the not-found case creates the
entry and other lookup failures are reported to the caller.
Made-with: Cursor
* fix(remote): use errors.Is for ErrNotFound in ensureMountDirectory
Replace fragile strings.Contains(lookupErr.Error(), ...) with
errors.Is(lookupErr, filer_pb.ErrNotFound) before calling CreateEntry.
Made-with: Cursor
* fix(remote): use LookupEntry so ErrNotFound is recognised after gRPC
Raw gRPC LookupDirectoryEntry returns a status error, not the sentinel,
so errors.Is(lookupErr, filer_pb.ErrNotFound) was always false. Use
filer_pb.LookupEntry which normalises not-found to ErrNotFound so the
mount directory is created when missing.
Made-with: Cursor
* test(remote): ignore weed shell banner in TestRemoteMountNoSync fs.ls count
Exclude master/filer and prompt lines from entry count so the assertion
checks only actual fs.ls output for empty -noSync mount.
Made-with: Cursor
* fix(remote.mount): use 0755 for mount dir, document bucket-less early return
Made-with: Cursor
* feat(remote.mount): replace -noSync with -metadataStrategy=lazy|eager
- Add -metadataStrategy flag (eager default, lazy skips upfront metadata pull)
- Accept lazy/eager case-insensitively; reject invalid values with clear error
- Rename TestRemoteMountNoSync to TestRemoteMountMetadataStrategyLazy
- Add TestRemoteMountMetadataStrategyEager and TestRemoteMountMetadataStrategyInvalid
Made-with: Cursor
* fix(remote.mount): validate strategy and remote before creating mount directory
Move strategy validation and validateMountRoot (lazy path) before
ensureMountDirectory so that invalid strategies or bad bucket/credentials
fail without leaving orphaned directory entries in the filer.
* refactor(remote.mount): remove unused remote param from ensureMountDirectory
The remote *RemoteStorageLocation parameter was left over from the old
syncMetadata signature. Only remoteConf.Name is used inside the function.
* doc(remote.mount): add TODO for HeadBucket-style validation
validateMountRoot currently lists all buckets to verify one exists.
Note the need for a targeted BucketExists method in the interface.
* refactor(remote.mount): use MetadataStrategy type and constants
Replace raw string comparisons with a MetadataStrategy type and
MetadataStrategyEager/MetadataStrategyLazy constants for clarity
and compile-time safety.
* refactor(remote.mount): rename MetadataStrategy to MetadataCacheStrategy
More precisely describes the purpose: controlling how metadata is
cached from the remote, not metadata handling in general.
* fix(remote.mount): remove validateMountRoot from lazy path
Lazy mount's purpose is to skip remote I/O. Validating via ListBuckets
contradicts that, especially on accounts with many buckets. Invalid
buckets or credentials will surface on first lazy access instead.
* fix(test): handle shell exit 0 in TestRemoteMountMetadataStrategyInvalid
The weed shell process exits with code 0 even when individual commands
fail — errors appear in stdout. Check output instead of requiring a
non-nil error.
* test(remote.mount): remove metadataStrategy shell integration tests
These tests only verify string output from a shell process that always
exits 0 — they cannot meaningfully validate eager vs lazy behavior
without a real remote backend.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* Fix remote.meta.sync TTL issue (#8021)
Remote entries should not have TTL applied because they represent files
in remote storage, not local SeaweedFS files. When TTL was configured on
a prefix, remote.meta.sync would create entries that immediately expired,
causing them to be deleted and recreated on each sync.
Changes:
- Set TtlSec=0 explicitly when creating remote entries in remote.meta.sync
- Skip TTL application in CreateEntry handler for entries with Remote field set
Fixes#8021
* Add TTL protection for remote entries in update path
- Set TtlSec=0 in doSaveRemoteEntry before calling UpdateEntry
- Add server-side TTL protection in UpdateEntry handler for remote entries
- Ensures remote entries don't inherit or preserve TTL when updated
* chore: execute goimports to format the code
Signed-off-by: promalert <promalert@outlook.com>
* goimports -w .
---------
Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Chris Lu
Peter Dodd
promalert
Aleksey Kosov
chrislu