605e41cadf
Update weed/replication/sink/azuresink/azure_sink.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
3 days ago
b3e2a03cbe
Address fourth round of Gemini Code Assist review comments
Fixed two critical issues identified in the fourth review:
1. HIGH: Handle BlobAlreadyExists in append blob creation
- Problem: If append blob already exists, Create() fails causing replication failure
- Fix: Added bloberror.HasCode(err, bloberror.BlobAlreadyExists) check
- Behavior: Existing append blobs are now acceptable, appends can proceed
- Impact: Makes replication sink more robust, prevents unnecessary failures
- Location: azure_sink.go CreateEntry function
2. MEDIUM: Configure custom retry policy for download resiliency
- Problem: Old SDK had MaxRetryRequests: 20, new SDK defaults to 3 retries
- Fix: Configured policy.RetryOptions with MaxRetries: 10
- Settings: TryTimeout=1min, RetryDelay=2s, MaxRetryDelay=1min
- Impact: Maintains similar resiliency in unreliable network conditions
- Location: azure_storage_client.go client initialization
Changes:
- Added import: github.com/Azure/azure-sdk-for-go/sdk/azcore/policy
- Updated NewClientWithSharedKeyCredential to include ClientOptions with retry policy
- Updated CreateEntry error handling to allow BlobAlreadyExists
Technical details:
- Retry policy uses exponential backoff (default SDK behavior)
- MaxRetries=10 provides good balance (was 20 in old SDK, default is 3)
- TryTimeout prevents individual requests from hanging indefinitely
- BlobAlreadyExists handling allows idempotent append operations
All tests pass. Build succeeds. Code is more resilient and robust.
3 days ago
b2d65168ab
Address third round of Gemini Code Assist review comments
Fixed all issues identified in the third review:
1. MEDIUM: Use bloberror.HasCode for ContainerAlreadyExists
- Replaced fragile string check with bloberror.HasCode()
- More robust and aligned with Azure SDK best practices
- Applied to CreateBucket test
2. MEDIUM: Use bloberror.HasCode for BlobNotFound in test
- Replaced generic error check with specific BlobNotFound check
- Makes test more precise and verifies correct error returned
- Applied to VerifyDeleted test
3. MEDIUM: Made DeleteEntry idempotent in azure_sink.go
- Now returns nil (no error) if blob doesn't exist
- Uses bloberror.HasCode(err, bloberror.BlobNotFound)
- Consistent with DeleteFile implementation
- Makes replication sink more robust to retries
Changes:
- Added import to azure_storage_client_test.go: bloberror
- Added import to azure_sink.go: bloberror
- Updated CreateBucket test to use bloberror.HasCode
- Updated VerifyDeleted test to use bloberror.HasCode
- Updated DeleteEntry to be idempotent
All tests pass. Build succeeds. Code uses Azure SDK best practices.
3 days ago
54b10f3351
Address Gemini Code Assist review comments
Fixed three issues identified by Gemini Code Assist:
1. HIGH: ReadFile now uses blob.CountToEnd when size is 0
- Old SDK: size=0 meant "read to end"
- New SDK: size=0 means "read 0 bytes"
- Fix: Use blob.CountToEnd (-1) to read entire blob from offset
2. MEDIUM: Use to.Ptr() instead of slice trick for DeleteSnapshots
- Replaced &[]Type{value}[0] with to.Ptr(value)
- Cleaner, more idiomatic Azure SDK pattern
- Applied to both azure_storage_client.go and azure_sink.go
3. Added missing imports:
- github.com/Azure/azure-sdk-for-go/sdk/azcore/to
These changes improve code clarity and correctness while following
Azure SDK best practices.
3 days ago
ff271aba16
Migrate from deprecated azure-storage-blob-go to modern Azure SDK
Migrates Azure Blob Storage integration from the deprecated
github.com/Azure/azure-storage-blob-go to the modern
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob SDK.
## Changes
### Removed Files
- weed/remote_storage/azure/azure_highlevel.go
- Custom upload helper no longer needed with new SDK
### Updated Files
- weed/remote_storage/azure/azure_storage_client.go
- Migrated from ServiceURL/ContainerURL/BlobURL to Client-based API
- Updated client creation using NewClientWithSharedKeyCredential
- Replaced ListBlobsFlatSegment with NewListBlobsFlatPager
- Updated Download to DownloadStream with proper HTTPRange
- Replaced custom uploadReaderAtToBlockBlob with UploadStream
- Updated GetProperties, SetMetadata, Delete to use new client methods
- Fixed metadata conversion to return map[string]*string
- weed/replication/sink/azuresink/azure_sink.go
- Migrated from ContainerURL to Client-based API
- Updated client initialization
- Replaced AppendBlobURL with AppendBlobClient
- Updated error handling to use azcore.ResponseError
- Added streaming.NopCloser for AppendBlock
### New Test Files
- weed/remote_storage/azure/azure_storage_client_test.go
- Comprehensive unit tests for all client operations
- Tests for Traverse, ReadFile, WriteFile, UpdateMetadata, Delete
- Tests for metadata conversion function
- Benchmark tests
- Integration tests (skippable without credentials)
- weed/replication/sink/azuresink/azure_sink_test.go
- Unit tests for Azure sink operations
- Tests for CreateEntry, UpdateEntry, DeleteEntry
- Tests for cleanKey function
- Tests for configuration-based initialization
- Integration tests (skippable without credentials)
- Benchmark tests
### Dependency Updates
- go.mod: Removed github.com/Azure/azure-storage-blob-go v0.15.0
- go.mod: Made github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.2 direct dependency
- All deprecated dependencies automatically cleaned up
## API Migration Summary
Old SDK → New SDK mappings:
- ServiceURL → Client (service-level operations)
- ContainerURL → ContainerClient
- BlobURL → BlobClient
- BlockBlobURL → BlockBlobClient
- AppendBlobURL → AppendBlobClient
- ListBlobsFlatSegment() → NewListBlobsFlatPager()
- Download() → DownloadStream()
- Upload() → UploadStream()
- Marker-based pagination → Pager-based pagination
- azblob.ResponseError → azcore.ResponseError
## Testing
All tests pass:
- ✅ Unit tests for metadata conversion
- ✅ Unit tests for helper functions (cleanKey)
- ✅ Interface implementation tests
- ✅ Build successful
- ✅ No compilation errors
- ✅ Integration tests available (require Azure credentials)
## Benefits
- ✅ Uses actively maintained SDK
- ✅ Better performance with modern API design
- ✅ Improved error handling
- ✅ Removes ~200 lines of custom upload code
- ✅ Reduces dependency count
- ✅ Better async/streaming support
- ✅ Future-proof against SDK deprecation
## Backward Compatibility
The changes are transparent to users:
- Same configuration parameters (account name, account key)
- Same functionality and behavior
- No changes to SeaweedFS API or user-facing features
- Existing Azure storage configurations continue to work
## Breaking Changes
None - this is an internal implementation change only.
3 days ago
283d9e0079
Add context with request ( #6824 )
5 months ago
ec155022e7
"golang.org/x/exp/slices" => "slices" and go fmt
10 months ago
7a75928e7d
Azure sink: avoid overwriting existing files
1 year ago
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
3 years ago
11f99836c3
filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0)
fix https://github.com/seaweedfs/seaweedfs/issues/3468
3 years ago
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
3 years ago
bff1ccc1de
fix compilation
3 years ago
7c39a18ba5
update azure library
4 years ago
45bffc92a8
filer.backup: fix cloud sinks when updating entry
4 years ago
540441fd38
go fmt
5 years ago
678c54d705
data sink: add incremental mode
5 years ago
a8624c2e4f
read from alternative replica
related to https://github.com/chrislusf/seaweedfs/issues/1512
5 years ago
387ab6796f
filer: cross cluster synchronization
5 years ago
eb7929a971
rename filer2 to filer
5 years ago
c27e18aa6a
read possible old deleted chunks
5 years ago
c647deace1
file size support set file length
use Attr.FileSize and TotalChunkSize to determine file size
5 years ago
97d97f3528
go code can read and write chunk manifest
5 years ago
f282ed444b
refactoring
6 years ago
ae2ee379c0
consistent 64bit size
6 years ago
2e3f6ad3a9
filer: remember content is gzipped or not
6 years ago
13e215ee5c
filer: option to encrypt data on volume server
6 years ago
892e726eb9
avoid reusing context object
fix https://github.com/chrislusf/seaweedfs/issues/1182
6 years ago
d335f04de6
support env variables to overwrite toml file
6 years ago
6fc6322c90
Change joeslay paths to chrislusf paths
6 years ago
595a1beff0
Swap imports to use joeslay
6 years ago
b3b42bc947
replicate need to include new entry path
7 years ago
55bab1b456
add context.Context
7 years ago
8dfac6a4cf
working b2 sink
7 years ago
6a756136ef
go fmt
7 years ago
b861899919
use released version of azblob
replace https://github.com/chrislusf/seaweedfs/pull/754
7 years ago
08266b7256
go fmt
7 years ago
9f4c2f87d3
adding filer replication sink to Azure
7 years ago
Chris Lu
Chris Lu
Aleksey Kosov
j.laycock
Chris Lu