From 70df119df573ce81f91cf0b3ef44e032edf16198 Mon Sep 17 00:00:00 2001 From: chrislu Date: Thu, 6 Nov 2025 23:06:16 -0800 Subject: [PATCH] Update README.md --- test/foundationdb/README.md | 41 ++++++++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/test/foundationdb/README.md b/test/foundationdb/README.md index fdc2c666b..fe0b884ca 100644 --- a/test/foundationdb/README.md +++ b/test/foundationdb/README.md @@ -61,12 +61,17 @@ The test environment includes: #### `foundationdb_integration_test.go` - Basic CRUD operations (Create, Read, Update, Delete) -- Directory operations and listing +- Directory operations and listing: + - `ListDirectoryEntries` - List all entries in a directory + - `ListDirectoryPrefixedEntries` - List entries matching a prefix + - `DeleteFolderChildren` - Bulk deletion of directory contents - Transaction handling (begin, commit, rollback) - Key-Value operations - Large entry handling with compression - Error scenarios and edge cases +**Note:** These tests operate at the filer store level, testing the metadata index operations that underpin S3 bucket listing and directory tree operations. + #### `foundationdb_concurrent_test.go` - Concurrent insert operations across multiple goroutines - Concurrent read/write operations on shared files @@ -75,6 +80,17 @@ The test environment includes: - Concurrent key-value operations - Stress testing under load +#### `test_fdb_s3.sh` - End-to-End S3 Integration Tests +- **S3 bucket creation** - Create buckets via S3 API +- **S3 file upload** - Upload files to buckets +- **S3 bucket listing** (`aws s3 ls`) - **Validates listing operations work correctly** +- **S3 file download** - Retrieve and verify file contents +- **S3 file deletion** - Delete objects and verify removal +- **FoundationDB backend verification** - Confirms data is stored in FDB +- **Filer directory operations** - Direct filer API calls for directory creation/listing + +**This test validates the complete S3 workflow including the listing operations that were problematic in earlier versions.** + #### Unit Tests (`weed/filer/foundationdb/foundationdb_store_test.go`) - Store initialization and configuration - Key generation and directory prefixes @@ -147,12 +163,29 @@ make check-env # Validate configuration make test # All tests make test-unit # Go unit tests make test-integration # Integration tests -make test-e2e # End-to-end S3 tests +make test-e2e # End-to-end S3 tests (includes S3 bucket listing) make test-crud # Basic CRUD operations make test-concurrent # Concurrency tests make test-benchmark # Performance benchmarks ``` +#### S3 and Listing Operation Coverage + +**✅ Currently Tested:** +- **S3 bucket listing** (`aws s3 ls`) - Validated in `test_fdb_s3.sh` +- **Directory metadata listing** (`ListDirectoryEntries`) - Tested in `foundationdb_integration_test.go` +- **Prefix-based listing** (`ListDirectoryPrefixedEntries`) - Tested in `foundationdb_integration_test.go` +- **Filer directory operations** - Basic filer API calls in `test_fdb_s3.sh` +- **Metadata index operations** - All CRUD operations on directory entries + +**⚠️ Limited/Future Coverage:** +- **Recursive tree operations** - Not explicitly tested (e.g., `weed filer.tree` command) +- **Large directory stress tests** - Listings with thousands of entries not currently benchmarked +- **Concurrent listing operations** - Multiple simultaneous directory listings under load +- **S3 ListObjectsV2 pagination** - Large bucket listing with continuation tokens + +**Recommendation:** If experiencing issues with S3 listing operations in production, add stress tests for large directories and concurrent listing scenarios to validate FoundationDB's range scan performance at scale. + ### Debug Commands ```bash @@ -268,7 +301,9 @@ docker-compose exec fdb-init fdbcli ### Listing Operations Return Empty Results -**Symptoms:** Uploads succeed, direct file reads work, but listing operations (s3.bucket.list, ls/tree) return no results. +**Symptoms:** Uploads succeed, direct file reads work, but listing operations (`aws s3 ls`, `s3.bucket.list`, `weed filer.ls/tree`) return no results. + +**Test Coverage:** The `test_fdb_s3.sh` script explicitly tests S3 bucket listing (`aws s3 ls`) to catch this class of issue. Integration tests cover the underlying `ListDirectoryEntries` operations. **Diagnostic steps:**