* s3: support STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER for signed chunked uploads with checksums
When AWS SDK v2 clients upload with both chunked encoding and checksum
validation enabled, they use the x-amz-content-sha256 header value of
STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER instead of the simpler
STREAMING-AWS4-HMAC-SHA256-PAYLOAD.
This caused the chunked reader to not be properly activated, resulting
in chunk-signature metadata being stored as part of the file content.
Changes:
- Add streamingSignedPayloadTrailer constant for the new header value
- Update isRequestSignStreamingV4() to recognize this header
- Update newChunkedReader() to handle this streaming type
- Update calculateSeedSignature() to accept this header
- Add unit test for signed streaming upload with trailer
Fixes issue where Quarkus/AWS SDK v2 uploads with checksum validation
resulted in corrupted file content containing chunk-signature data.
* address review comments: add trailer signature to test, fix constant alignment
* test: separate canonical trailer text (\n) from on-wire format (\r\n)
* test: add negative test for invalid trailer signature
* refactor: check HTTP method first in streaming auth checks (fail-fast)
* test: handle crc32 Write error return for completeness
* refactor: extract createTrailerStreamingRequest helper to reduce test duplication
* fmt
* docs: clarify test comment about trailer signature validation status
* refactor: calculate chunk data length dynamically instead of hardcoding
* Update weed/s3api/chunked_reader_v4_test.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix: use current time for signatures instead of hardcoded past date
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Add nginx reverse proxy documentation for S3 API
Fixes#7407
Add comprehensive documentation and example configuration for using
nginx as a reverse proxy with SeaweedFS S3 API while maintaining AWS
Signature V4 authentication compatibility.
Changes:
- Add docker/nginx/README.md with detailed setup guide
- Add docker/nginx/s3-example.conf with working configuration
- Update docker/nginx/proxy.conf with important S3 notes
The documentation covers:
- Critical requirements for AWS Signature V4 authentication
- Common mistakes and why they break S3 authentication
- Complete working nginx configurations
- Debugging tips and troubleshooting
- Performance tuning recommendations
* Fix IPv6 host header formatting to match AWS SDK behavior
Follow-up to PR #7403
When a default port (80 for HTTP, 443 for HTTPS) is stripped from an
IPv6 address, the square brackets should also be removed to match AWS
SDK behavior for S3 signature calculation.
Reference: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/internal/v4/host.go
The AWS SDK's stripPort function explicitly removes brackets when
returning an IPv6 address without a port.
Changes:
- Update extractHostHeader to strip brackets from IPv6 addresses when
no port or default port is used
- Update test expectations to match AWS SDK behavior
- Add detailed comments explaining the AWS SDK compatibility requirement
This ensures S3 signature validation works correctly with IPv6 addresses
behind reverse proxies, matching AWS S3 canonical request format.
Fixes the issue raised in PR #7403 comment:
https://github.com/seaweedfs/seaweedfs/pull/7403#issuecomment-3471105438
* Update docker/nginx/README.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Add nginx reverse proxy documentation for S3 API
Fixes#7407
Add comprehensive documentation and example configuration for using
nginx as a reverse proxy with SeaweedFS S3 API while maintaining AWS
Signature V4 authentication compatibility.
Changes:
- Add docker/nginx/README.md with detailed setup guide
- Add docker/nginx/s3-example.conf with working configuration
- Update docker/nginx/proxy.conf with important S3 notes
The documentation covers:
- Critical requirements for AWS Signature V4 authentication
- Common mistakes and why they break S3 authentication
- Complete working nginx configurations
- Debugging tips and troubleshooting
- Performance tuning recommendations
Fix IPv6 host header formatting to match AWS SDK behavior
Follow-up to PR #7403
When a default port (80 for HTTP, 443 for HTTPS) is stripped from an
IPv6 address, the square brackets should also be removed to match AWS
SDK behavior for S3 signature calculation.
Reference: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/internal/v4/host.go
The AWS SDK's stripPort function explicitly removes brackets when
returning an IPv6 address without a port.
Changes:
- Update extractHostHeader to strip brackets from IPv6 addresses when
no port or default port is used
- Update test expectations to match AWS SDK behavior
- Add detailed comments explaining the AWS SDK compatibility requirement
This ensures S3 signature validation works correctly with IPv6 addresses
behind reverse proxies, matching AWS S3 canonical request format.
Fixes the issue raised in PR #7403 comment:
https://github.com/seaweedfs/seaweedfs/pull/7403#issuecomment-3471105438
* Revert "Merge branch 'fix-ipv6-brackets-default-port' of https://github.com/seaweedfs/seaweedfs into fix-ipv6-brackets-default-port"
This reverts commit cca3f3985f, reversing
changes made to 2b8f9de78e.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* * Fix s3 auth with proxy request
* * 6649 Add unit test for signature v4
* address comments
* fix for tests
* ipv6
* address comments
* setting scheme
Works for both cases (direct HTTPS and behind proxy)
* trim for ipv6
* Corrected Scheme Precedence Order
* trim
* accurate
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Signature verification should not check permissions - that's done later in authRequest
* test permissions during signature verfication
* fix s3 test path
* s3tests_boto3 => s3tests
* remove extra lines
This change is caching HMAC hashers for repeated use in subsequent
requests and chunks, so they don't have to be initialized from
scratch every time.
On my local computer this gives me ~5-6 times faster signature
calculation and ~5-6.5% more throughput in S3 requests. The smaller
the payload the better the throughput gets.
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Chris Lu
chrislu
zuzuviewer
Tom Crasset
Joon Young Baik
Alexey
steve.wei
Er2
clonefetch
Patrick Schmidt
Ryan Russell
Eng Zer Jun
Konstantin Lebedev