Browse Source
S3 API: Add SSE-C (#7143)
S3 API: Add SSE-C (#7143)
* implement sse-c * fix Content-Range * adding tests * Update s3_sse_c_test.go * copy sse-c objects * adding tests * refactor * multi reader * remove extra write header call * refactor * SSE-C encrypted objects do not support HTTP Range requests * robust * fix server starts * Update Makefile * Update Makefile * ci: remove SSE-C integration tests and workflows; delete test/s3/encryption/ * s3: SSE-C MD5 must be base64 (case-sensitive); fix validation, comparisons, metadata storage; update tests * minor * base64 * Update SSE-C_IMPLEMENTATION.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update SSE-C_IMPLEMENTATION.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * address comments * fix test * fix compilation --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>pull/7149/head
committed by
GitHub
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 1265 additions and 21 deletions
-
4.github/workflows/s3-go-tests.yml
-
1.gitignore
-
169SSE-C_IMPLEMENTATION.md
-
11weed/s3api/s3_constants/header.go
-
275weed/s3api/s3_sse_c.go
-
63weed/s3api/s3_sse_c_range_test.go
-
412weed/s3api/s3_sse_c_test.go
-
134weed/s3api/s3api_object_handlers.go
-
155weed/s3api/s3api_object_handlers_copy.go
-
19weed/s3api/s3api_object_handlers_put.go
-
34weed/s3api/s3err/s3api_errors.go
-
9weed/server/filer_server_handlers_write_autochunk.go
@ -0,0 +1,169 @@ |
|||||
|
# Server-Side Encryption with Customer-Provided Keys (SSE-C) Implementation |
||||
|
|
||||
|
This document describes the implementation of SSE-C support in SeaweedFS, addressing the feature request from [GitHub Discussion #5361](https://github.com/seaweedfs/seaweedfs/discussions/5361). |
||||
|
|
||||
|
## Overview |
||||
|
|
||||
|
SSE-C allows clients to provide their own encryption keys for server-side encryption of objects stored in SeaweedFS. The server encrypts the data using the customer-provided AES-256 key but does not store the key itself - only an MD5 hash of the key for validation purposes. |
||||
|
|
||||
|
## Implementation Details |
||||
|
|
||||
|
### Architecture |
||||
|
|
||||
|
The SSE-C implementation follows a transparent encryption/decryption pattern: |
||||
|
|
||||
|
1. **Upload (PUT/POST)**: Data is encrypted with the customer key before being stored |
||||
|
2. **Download (GET/HEAD)**: Encrypted data is decrypted on-the-fly using the customer key |
||||
|
3. **Metadata Storage**: Only the encryption algorithm and key MD5 are stored as metadata |
||||
|
|
||||
|
### Key Components |
||||
|
|
||||
|
#### 1. Constants and Headers (`weed/s3api/s3_constants/header.go`) |
||||
|
- Added AWS-compatible SSE-C header constants |
||||
|
- Support for both regular and copy-source SSE-C headers |
||||
|
|
||||
|
#### 2. Core SSE-C Logic (`weed/s3api/s3_sse_c.go`) |
||||
|
- **SSECustomerKey**: Structure to hold customer encryption key and metadata |
||||
|
- **SSECEncryptedReader**: Streaming encryption with AES-256-CTR mode |
||||
|
- **SSECDecryptedReader**: Streaming decryption with IV extraction |
||||
|
- **validateAndParseSSECHeaders**: Shared validation logic (DRY principle) |
||||
|
- **ParseSSECHeaders**: Parse regular SSE-C headers |
||||
|
- **ParseSSECCopySourceHeaders**: Parse copy-source SSE-C headers |
||||
|
- Header validation and parsing functions |
||||
|
- Metadata extraction and response handling |
||||
|
|
||||
|
#### 3. Error Handling (`weed/s3api/s3err/s3api_errors.go`) |
||||
|
- New error codes for SSE-C validation failures |
||||
|
- AWS-compatible error messages and HTTP status codes |
||||
|
|
||||
|
#### 4. S3 API Integration |
||||
|
- **PUT Object Handler**: Encrypts data streams transparently |
||||
|
- **GET Object Handler**: Decrypts data streams transparently |
||||
|
- **HEAD Object Handler**: Validates keys and returns appropriate headers |
||||
|
- **Metadata Storage**: Integrates with existing `SaveAmzMetaData` function |
||||
|
|
||||
|
### Encryption Scheme |
||||
|
|
||||
|
- **Algorithm**: AES-256-CTR (Counter mode) |
||||
|
- **Key Size**: 256 bits (32 bytes) |
||||
|
- **IV Generation**: Random 16-byte IV per object |
||||
|
- **Storage Format**: `[IV][EncryptedData]` where IV is prepended to encrypted content |
||||
|
|
||||
|
### Metadata Storage |
||||
|
|
||||
|
SSE-C metadata is stored in the filer's extended attributes: |
||||
|
``` |
||||
|
x-amz-server-side-encryption-customer-algorithm: "AES256" |
||||
|
x-amz-server-side-encryption-customer-key-md5: "<md5-hash-of-key>" |
||||
|
``` |
||||
|
|
||||
|
## API Compatibility |
||||
|
|
||||
|
### Required Headers for Encryption (PUT/POST) |
||||
|
``` |
||||
|
x-amz-server-side-encryption-customer-algorithm: AES256 |
||||
|
x-amz-server-side-encryption-customer-key: <base64-encoded-256-bit-key> |
||||
|
x-amz-server-side-encryption-customer-key-md5: <md5-hash-of-key> |
||||
|
``` |
||||
|
|
||||
|
### Required Headers for Decryption (GET/HEAD) |
||||
|
Same headers as encryption - the server validates the key MD5 matches. |
||||
|
|
||||
|
### Copy Operations |
||||
|
Support for copy-source SSE-C headers: |
||||
|
``` |
||||
|
x-amz-copy-source-server-side-encryption-customer-algorithm |
||||
|
x-amz-copy-source-server-side-encryption-customer-key |
||||
|
x-amz-copy-source-server-side-encryption-customer-key-md5 |
||||
|
``` |
||||
|
|
||||
|
## Error Handling |
||||
|
|
||||
|
The implementation provides AWS-compatible error responses: |
||||
|
|
||||
|
- **InvalidEncryptionAlgorithmError**: Non-AES256 algorithm specified |
||||
|
- **InvalidArgument**: Invalid key format, size, or MD5 mismatch |
||||
|
- **Missing customer key**: Object encrypted but no key provided |
||||
|
- **Unnecessary customer key**: Object not encrypted but key provided |
||||
|
|
||||
|
## Security Considerations |
||||
|
|
||||
|
1. **Key Management**: Customer keys are never stored - only MD5 hashes for validation |
||||
|
2. **IV Randomness**: Fresh random IV generated for each object |
||||
|
3. **Transparent Security**: Volume servers never see unencrypted data |
||||
|
4. **Key Validation**: Strict validation of key format, size, and MD5 |
||||
|
|
||||
|
## Testing |
||||
|
|
||||
|
Comprehensive test suite covers: |
||||
|
- Header validation and parsing (regular and copy-source) |
||||
|
- Encryption/decryption round-trip |
||||
|
- Error condition handling |
||||
|
- Metadata extraction |
||||
|
- Code reuse validation (DRY principle) |
||||
|
- AWS S3 compatibility |
||||
|
|
||||
|
Run tests with: |
||||
|
```bash |
||||
|
go test -v ./weed/s3api |
||||
|
|
||||
|
## Usage Example |
||||
|
|
||||
|
### Upload with SSE-C |
||||
|
```bash |
||||
|
# Generate a 256-bit key |
||||
|
KEY=$(openssl rand -base64 32) |
||||
|
KEY_MD5=$(echo -n "$KEY" | base64 -d | openssl dgst -md5 -binary | base64) |
||||
|
|
||||
|
# Upload object with SSE-C |
||||
|
curl -X PUT "http://localhost:8333/bucket/object" \ |
||||
|
-H "x-amz-server-side-encryption-customer-algorithm: AES256" \ |
||||
|
-H "x-amz-server-side-encryption-customer-key: $KEY" \ |
||||
|
-H "x-amz-server-side-encryption-customer-key-md5: $KEY_MD5" \ |
||||
|
--data-binary @file.txt |
||||
|
``` |
||||
|
|
||||
|
### Download with SSE-C |
||||
|
```bash |
||||
|
# Download object with SSE-C (same key required) |
||||
|
curl "http://localhost:8333/bucket/object" \ |
||||
|
-H "x-amz-server-side-encryption-customer-algorithm: AES256" \ |
||||
|
-H "x-amz-server-side-encryption-customer-key: $KEY" \ |
||||
|
-H "x-amz-server-side-encryption-customer-key-md5: $KEY_MD5" |
||||
|
``` |
||||
|
|
||||
|
## Integration Points |
||||
|
|
||||
|
### Existing SeaweedFS Features |
||||
|
- **Filer Metadata**: Extends existing metadata storage |
||||
|
- **Volume Servers**: No changes required - store encrypted data transparently |
||||
|
- **S3 API**: Integrates seamlessly with existing handlers |
||||
|
- **Versioning**: Compatible with object versioning |
||||
|
- **Multipart Upload**: Ready for multipart upload integration |
||||
|
|
||||
|
### Future Enhancements |
||||
|
- **SSE-S3**: Server-managed encryption keys |
||||
|
- **SSE-KMS**: External key management service integration |
||||
|
- **Performance Optimization**: Hardware acceleration for encryption |
||||
|
- **Compliance**: Enhanced audit logging for encrypted objects |
||||
|
|
||||
|
## File Changes Summary |
||||
|
|
||||
|
1. **`weed/s3api/s3_constants/header.go`** - Added SSE-C header constants |
||||
|
2. **`weed/s3api/s3_sse_c.go`** - Core SSE-C implementation (NEW) |
||||
|
3. **`weed/s3api/s3_sse_c_test.go`** - Comprehensive test suite (NEW) |
||||
|
4. **`weed/s3api/s3err/s3api_errors.go`** - Added SSE-C error codes |
||||
|
5. **`weed/s3api/s3api_object_handlers.go`** - GET/HEAD with SSE-C support |
||||
|
6. **`weed/s3api/s3api_object_handlers_put.go`** - PUT with SSE-C support |
||||
|
7. **`weed/server/filer_server_handlers_write_autochunk.go`** - Metadata storage |
||||
|
|
||||
|
## Compliance |
||||
|
|
||||
|
This implementation follows the [AWS S3 SSE-C specification](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html) for maximum compatibility with existing S3 clients and tools. |
||||
|
|
||||
|
## Performance Impact |
||||
|
|
||||
|
- **Encryption Overhead**: Minimal CPU impact with efficient AES-CTR streaming |
||||
|
- **Memory Usage**: Constant memory usage via streaming encryption/decryption |
||||
|
- **Storage Overhead**: 16 bytes per object for IV storage |
||||
|
- **Network**: No additional network overhead |
@ -0,0 +1,275 @@ |
|||||
|
package s3api |
||||
|
|
||||
|
import ( |
||||
|
"bytes" |
||||
|
"crypto/aes" |
||||
|
"crypto/cipher" |
||||
|
"crypto/md5" |
||||
|
"crypto/rand" |
||||
|
"encoding/base64" |
||||
|
"errors" |
||||
|
"fmt" |
||||
|
"io" |
||||
|
"net/http" |
||||
|
|
||||
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" |
||||
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err" |
||||
|
) |
||||
|
|
||||
|
const ( |
||||
|
// SSE-C constants
|
||||
|
SSECustomerAlgorithmAES256 = "AES256" |
||||
|
SSECustomerKeySize = 32 // 256 bits
|
||||
|
AESBlockSize = 16 // AES block size in bytes
|
||||
|
) |
||||
|
|
||||
|
// SSE-C related errors
|
||||
|
var ( |
||||
|
ErrInvalidRequest = errors.New("invalid request") |
||||
|
ErrInvalidEncryptionAlgorithm = errors.New("invalid encryption algorithm") |
||||
|
ErrInvalidEncryptionKey = errors.New("invalid encryption key") |
||||
|
ErrSSECustomerKeyMD5Mismatch = errors.New("customer key MD5 mismatch") |
||||
|
ErrSSECustomerKeyMissing = errors.New("customer key missing") |
||||
|
ErrSSECustomerKeyNotNeeded = errors.New("customer key not needed") |
||||
|
) |
||||
|
|
||||
|
// SSECustomerKey represents a customer-provided encryption key for SSE-C
|
||||
|
type SSECustomerKey struct { |
||||
|
Algorithm string |
||||
|
Key []byte |
||||
|
KeyMD5 string |
||||
|
} |
||||
|
|
||||
|
// SSECDecryptedReader wraps an io.Reader to provide SSE-C decryption
|
||||
|
type SSECDecryptedReader struct { |
||||
|
reader io.Reader |
||||
|
cipher cipher.Stream |
||||
|
customerKey *SSECustomerKey |
||||
|
first bool |
||||
|
} |
||||
|
|
||||
|
// IsSSECRequest checks if the request contains SSE-C headers
|
||||
|
func IsSSECRequest(r *http.Request) bool { |
||||
|
return r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerAlgorithm) != "" |
||||
|
} |
||||
|
|
||||
|
// validateAndParseSSECHeaders does the core validation and parsing logic
|
||||
|
func validateAndParseSSECHeaders(algorithm, key, keyMD5 string) (*SSECustomerKey, error) { |
||||
|
if algorithm == "" && key == "" && keyMD5 == "" { |
||||
|
return nil, nil // No SSE-C headers
|
||||
|
} |
||||
|
|
||||
|
if algorithm == "" || key == "" || keyMD5 == "" { |
||||
|
return nil, ErrInvalidRequest |
||||
|
} |
||||
|
|
||||
|
if algorithm != SSECustomerAlgorithmAES256 { |
||||
|
return nil, ErrInvalidEncryptionAlgorithm |
||||
|
} |
||||
|
|
||||
|
// Decode and validate key
|
||||
|
keyBytes, err := base64.StdEncoding.DecodeString(key) |
||||
|
if err != nil { |
||||
|
return nil, ErrInvalidEncryptionKey |
||||
|
} |
||||
|
|
||||
|
if len(keyBytes) != SSECustomerKeySize { |
||||
|
return nil, ErrInvalidEncryptionKey |
||||
|
} |
||||
|
|
||||
|
// Validate key MD5 (base64-encoded MD5 of the raw key bytes; case-sensitive)
|
||||
|
sum := md5.Sum(keyBytes) |
||||
|
expectedMD5 := base64.StdEncoding.EncodeToString(sum[:]) |
||||
|
if keyMD5 != expectedMD5 { |
||||
|
return nil, ErrSSECustomerKeyMD5Mismatch |
||||
|
} |
||||
|
|
||||
|
return &SSECustomerKey{ |
||||
|
Algorithm: algorithm, |
||||
|
Key: keyBytes, |
||||
|
KeyMD5: keyMD5, |
||||
|
}, nil |
||||
|
} |
||||
|
|
||||
|
// ValidateSSECHeaders validates SSE-C headers in the request
|
||||
|
func ValidateSSECHeaders(r *http.Request) error { |
||||
|
algorithm := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerAlgorithm) |
||||
|
key := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerKey) |
||||
|
keyMD5 := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerKeyMD5) |
||||
|
|
||||
|
_, err := validateAndParseSSECHeaders(algorithm, key, keyMD5) |
||||
|
return err |
||||
|
} |
||||
|
|
||||
|
// ParseSSECHeaders parses and validates SSE-C headers from the request
|
||||
|
func ParseSSECHeaders(r *http.Request) (*SSECustomerKey, error) { |
||||
|
algorithm := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerAlgorithm) |
||||
|
key := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerKey) |
||||
|
keyMD5 := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerKeyMD5) |
||||
|
|
||||
|
return validateAndParseSSECHeaders(algorithm, key, keyMD5) |
||||
|
} |
||||
|
|
||||
|
// ParseSSECCopySourceHeaders parses and validates SSE-C copy source headers from the request
|
||||
|
func ParseSSECCopySourceHeaders(r *http.Request) (*SSECustomerKey, error) { |
||||
|
algorithm := r.Header.Get(s3_constants.AmzCopySourceServerSideEncryptionCustomerAlgorithm) |
||||
|
key := r.Header.Get(s3_constants.AmzCopySourceServerSideEncryptionCustomerKey) |
||||
|
keyMD5 := r.Header.Get(s3_constants.AmzCopySourceServerSideEncryptionCustomerKeyMD5) |
||||
|
|
||||
|
return validateAndParseSSECHeaders(algorithm, key, keyMD5) |
||||
|
} |
||||
|
|
||||
|
// CreateSSECEncryptedReader creates a new encrypted reader for SSE-C
|
||||
|
func CreateSSECEncryptedReader(r io.Reader, customerKey *SSECustomerKey) (io.Reader, error) { |
||||
|
if customerKey == nil { |
||||
|
return r, nil |
||||
|
} |
||||
|
|
||||
|
// Create AES cipher
|
||||
|
block, err := aes.NewCipher(customerKey.Key) |
||||
|
if err != nil { |
||||
|
return nil, fmt.Errorf("failed to create AES cipher: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Generate random IV
|
||||
|
iv := make([]byte, AESBlockSize) |
||||
|
if _, err := io.ReadFull(rand.Reader, iv); err != nil { |
||||
|
return nil, fmt.Errorf("failed to generate IV: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Create CTR mode cipher
|
||||
|
stream := cipher.NewCTR(block, iv) |
||||
|
|
||||
|
// The encrypted stream is the IV (initialization vector) followed by the encrypted data.
|
||||
|
// The IV is randomly generated for each encryption operation and must be unique and unpredictable.
|
||||
|
// This is critical for the security of AES-CTR mode: reusing an IV with the same key breaks confidentiality.
|
||||
|
// By prepending the IV to the ciphertext, the decryptor can extract the IV to initialize the cipher.
|
||||
|
// Note: AES-CTR provides confidentiality only; use an additional MAC if integrity is required.
|
||||
|
// We model this with an io.MultiReader (IV first) and a cipher.StreamReader (encrypted payload).
|
||||
|
return io.MultiReader(bytes.NewReader(iv), &cipher.StreamReader{S: stream, R: r}), nil |
||||
|
} |
||||
|
|
||||
|
// CreateSSECDecryptedReader creates a new decrypted reader for SSE-C
|
||||
|
func CreateSSECDecryptedReader(r io.Reader, customerKey *SSECustomerKey) (io.Reader, error) { |
||||
|
if customerKey == nil { |
||||
|
return r, nil |
||||
|
} |
||||
|
|
||||
|
return &SSECDecryptedReader{ |
||||
|
reader: r, |
||||
|
customerKey: customerKey, |
||||
|
cipher: nil, // Will be initialized when we read the IV
|
||||
|
first: true, |
||||
|
}, nil |
||||
|
} |
||||
|
|
||||
|
// Read implements io.Reader for SSECDecryptedReader
|
||||
|
func (r *SSECDecryptedReader) Read(p []byte) (n int, err error) { |
||||
|
if r.first { |
||||
|
// First read: extract IV and initialize cipher
|
||||
|
r.first = false |
||||
|
iv := make([]byte, AESBlockSize) |
||||
|
|
||||
|
// Read IV from the beginning of the data
|
||||
|
_, err = io.ReadFull(r.reader, iv) |
||||
|
if err != nil { |
||||
|
return 0, fmt.Errorf("failed to read IV: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Create cipher with the extracted IV
|
||||
|
block, err := aes.NewCipher(r.customerKey.Key) |
||||
|
if err != nil { |
||||
|
return 0, fmt.Errorf("failed to create AES cipher: %v", err) |
||||
|
} |
||||
|
r.cipher = cipher.NewCTR(block, iv) |
||||
|
} |
||||
|
|
||||
|
// Decrypt data
|
||||
|
n, err = r.reader.Read(p) |
||||
|
if n > 0 { |
||||
|
r.cipher.XORKeyStream(p[:n], p[:n]) |
||||
|
} |
||||
|
return n, err |
||||
|
} |
||||
|
|
||||
|
// GetSourceSSECInfo extracts SSE-C information from source object metadata
|
||||
|
func GetSourceSSECInfo(metadata map[string][]byte) (algorithm string, keyMD5 string, isEncrypted bool) { |
||||
|
if alg, exists := metadata[s3_constants.AmzServerSideEncryptionCustomerAlgorithm]; exists { |
||||
|
algorithm = string(alg) |
||||
|
} |
||||
|
if md5, exists := metadata[s3_constants.AmzServerSideEncryptionCustomerKeyMD5]; exists { |
||||
|
keyMD5 = string(md5) |
||||
|
} |
||||
|
isEncrypted = algorithm != "" && keyMD5 != "" |
||||
|
return |
||||
|
} |
||||
|
|
||||
|
// CanDirectCopySSEC determines if we can directly copy chunks without decrypt/re-encrypt
|
||||
|
func CanDirectCopySSEC(srcMetadata map[string][]byte, copySourceKey *SSECustomerKey, destKey *SSECustomerKey) bool { |
||||
|
_, srcKeyMD5, srcEncrypted := GetSourceSSECInfo(srcMetadata) |
||||
|
|
||||
|
// Case 1: Source unencrypted, destination unencrypted -> Direct copy
|
||||
|
if !srcEncrypted && destKey == nil { |
||||
|
return true |
||||
|
} |
||||
|
|
||||
|
// Case 2: Source encrypted, same key for decryption and destination -> Direct copy
|
||||
|
if srcEncrypted && copySourceKey != nil && destKey != nil { |
||||
|
// Same key if MD5 matches exactly (base64 encoding is case-sensitive)
|
||||
|
return copySourceKey.KeyMD5 == srcKeyMD5 && |
||||
|
destKey.KeyMD5 == srcKeyMD5 |
||||
|
} |
||||
|
|
||||
|
// All other cases require decrypt/re-encrypt
|
||||
|
return false |
||||
|
} |
||||
|
|
||||
|
// SSECCopyStrategy represents the strategy for copying SSE-C objects
|
||||
|
type SSECCopyStrategy int |
||||
|
|
||||
|
const ( |
||||
|
SSECCopyDirect SSECCopyStrategy = iota // Direct chunk copy (fast)
|
||||
|
SSECCopyReencrypt // Decrypt and re-encrypt (slow)
|
||||
|
) |
||||
|
|
||||
|
// DetermineSSECCopyStrategy determines the optimal copy strategy
|
||||
|
func DetermineSSECCopyStrategy(srcMetadata map[string][]byte, copySourceKey *SSECustomerKey, destKey *SSECustomerKey) (SSECCopyStrategy, error) { |
||||
|
_, srcKeyMD5, srcEncrypted := GetSourceSSECInfo(srcMetadata) |
||||
|
|
||||
|
// Validate source key if source is encrypted
|
||||
|
if srcEncrypted { |
||||
|
if copySourceKey == nil { |
||||
|
return SSECCopyReencrypt, ErrSSECustomerKeyMissing |
||||
|
} |
||||
|
if copySourceKey.KeyMD5 != srcKeyMD5 { |
||||
|
return SSECCopyReencrypt, ErrSSECustomerKeyMD5Mismatch |
||||
|
} |
||||
|
} else if copySourceKey != nil { |
||||
|
// Source not encrypted but copy source key provided
|
||||
|
return SSECCopyReencrypt, ErrSSECustomerKeyNotNeeded |
||||
|
} |
||||
|
|
||||
|
if CanDirectCopySSEC(srcMetadata, copySourceKey, destKey) { |
||||
|
return SSECCopyDirect, nil |
||||
|
} |
||||
|
|
||||
|
return SSECCopyReencrypt, nil |
||||
|
} |
||||
|
|
||||
|
// MapSSECErrorToS3Error maps SSE-C custom errors to S3 API error codes
|
||||
|
func MapSSECErrorToS3Error(err error) s3err.ErrorCode { |
||||
|
switch err { |
||||
|
case ErrInvalidEncryptionAlgorithm: |
||||
|
return s3err.ErrInvalidEncryptionAlgorithm |
||||
|
case ErrInvalidEncryptionKey: |
||||
|
return s3err.ErrInvalidEncryptionKey |
||||
|
case ErrSSECustomerKeyMD5Mismatch: |
||||
|
return s3err.ErrSSECustomerKeyMD5Mismatch |
||||
|
case ErrSSECustomerKeyMissing: |
||||
|
return s3err.ErrSSECustomerKeyMissing |
||||
|
case ErrSSECustomerKeyNotNeeded: |
||||
|
return s3err.ErrSSECustomerKeyNotNeeded |
||||
|
default: |
||||
|
return s3err.ErrInvalidRequest |
||||
|
} |
||||
|
} |
@ -0,0 +1,63 @@ |
|||||
|
package s3api |
||||
|
|
||||
|
import ( |
||||
|
"bytes" |
||||
|
"crypto/md5" |
||||
|
"encoding/base64" |
||||
|
"io" |
||||
|
"net/http" |
||||
|
"net/http/httptest" |
||||
|
"testing" |
||||
|
|
||||
|
"github.com/gorilla/mux" |
||||
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" |
||||
|
) |
||||
|
|
||||
|
// ResponseRecorder that also implements http.Flusher
|
||||
|
type recorderFlusher struct{ *httptest.ResponseRecorder } |
||||
|
|
||||
|
func (r recorderFlusher) Flush() {} |
||||
|
|
||||
|
// TestSSECRangeRequestsNotSupported verifies that HTTP Range requests are rejected
|
||||
|
// for SSE-C encrypted objects because the IV is required at the beginning of the stream
|
||||
|
func TestSSECRangeRequestsNotSupported(t *testing.T) { |
||||
|
// Create a request with Range header and valid SSE-C headers
|
||||
|
req := httptest.NewRequest(http.MethodGet, "/b/o", nil) |
||||
|
req.Header.Set("Range", "bytes=10-20") |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerAlgorithm, "AES256") |
||||
|
|
||||
|
key := make([]byte, 32) |
||||
|
for i := range key { |
||||
|
key[i] = byte(i) |
||||
|
} |
||||
|
s := md5.Sum(key) |
||||
|
keyMD5 := base64.StdEncoding.EncodeToString(s[:]) |
||||
|
|
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKey, base64.StdEncoding.EncodeToString(key)) |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKeyMD5, keyMD5) |
||||
|
|
||||
|
// Attach mux vars to avoid panic in error writer
|
||||
|
req = mux.SetURLVars(req, map[string]string{"bucket": "b", "object": "o"}) |
||||
|
|
||||
|
// Create a mock HTTP response that simulates SSE-C encrypted object metadata
|
||||
|
proxyResponse := &http.Response{ |
||||
|
StatusCode: 200, |
||||
|
Header: make(http.Header), |
||||
|
Body: io.NopCloser(bytes.NewReader([]byte("mock encrypted data"))), |
||||
|
} |
||||
|
proxyResponse.Header.Set(s3_constants.AmzServerSideEncryptionCustomerAlgorithm, "AES256") |
||||
|
proxyResponse.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKeyMD5, keyMD5) |
||||
|
|
||||
|
// Call the function under test
|
||||
|
s3a := &S3ApiServer{} |
||||
|
rec := httptest.NewRecorder() |
||||
|
w := recorderFlusher{rec} |
||||
|
statusCode, _ := s3a.handleSSECResponse(req, proxyResponse, w) |
||||
|
|
||||
|
if statusCode != http.StatusRequestedRangeNotSatisfiable { |
||||
|
t.Fatalf("expected status %d, got %d", http.StatusRequestedRangeNotSatisfiable, statusCode) |
||||
|
} |
||||
|
if rec.Result().StatusCode != http.StatusRequestedRangeNotSatisfiable { |
||||
|
t.Fatalf("writer status expected %d, got %d", http.StatusRequestedRangeNotSatisfiable, rec.Result().StatusCode) |
||||
|
} |
||||
|
} |
@ -0,0 +1,412 @@ |
|||||
|
package s3api |
||||
|
|
||||
|
import ( |
||||
|
"bytes" |
||||
|
"crypto/md5" |
||||
|
"encoding/base64" |
||||
|
"fmt" |
||||
|
"io" |
||||
|
"net/http" |
||||
|
"testing" |
||||
|
|
||||
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" |
||||
|
) |
||||
|
|
||||
|
func base64MD5(b []byte) string { |
||||
|
s := md5.Sum(b) |
||||
|
return base64.StdEncoding.EncodeToString(s[:]) |
||||
|
} |
||||
|
|
||||
|
func TestSSECHeaderValidation(t *testing.T) { |
||||
|
// Test valid SSE-C headers
|
||||
|
req := &http.Request{Header: make(http.Header)} |
||||
|
|
||||
|
key := make([]byte, 32) // 256-bit key
|
||||
|
for i := range key { |
||||
|
key[i] = byte(i) |
||||
|
} |
||||
|
|
||||
|
keyBase64 := base64.StdEncoding.EncodeToString(key) |
||||
|
md5sum := md5.Sum(key) |
||||
|
keyMD5 := base64.StdEncoding.EncodeToString(md5sum[:]) |
||||
|
|
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerAlgorithm, "AES256") |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKey, keyBase64) |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKeyMD5, keyMD5) |
||||
|
|
||||
|
// Test validation
|
||||
|
err := ValidateSSECHeaders(req) |
||||
|
if err != nil { |
||||
|
t.Errorf("Expected valid headers, got error: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Test parsing
|
||||
|
customerKey, err := ParseSSECHeaders(req) |
||||
|
if err != nil { |
||||
|
t.Errorf("Expected successful parsing, got error: %v", err) |
||||
|
} |
||||
|
|
||||
|
if customerKey == nil { |
||||
|
t.Error("Expected customer key, got nil") |
||||
|
} |
||||
|
|
||||
|
if customerKey.Algorithm != "AES256" { |
||||
|
t.Errorf("Expected algorithm AES256, got %s", customerKey.Algorithm) |
||||
|
} |
||||
|
|
||||
|
if !bytes.Equal(customerKey.Key, key) { |
||||
|
t.Error("Key doesn't match original") |
||||
|
} |
||||
|
|
||||
|
if customerKey.KeyMD5 != keyMD5 { |
||||
|
t.Errorf("Expected key MD5 %s, got %s", keyMD5, customerKey.KeyMD5) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestSSECCopySourceHeaders(t *testing.T) { |
||||
|
// Test valid SSE-C copy source headers
|
||||
|
req := &http.Request{Header: make(http.Header)} |
||||
|
|
||||
|
key := make([]byte, 32) // 256-bit key
|
||||
|
for i := range key { |
||||
|
key[i] = byte(i) + 1 // Different from regular test
|
||||
|
} |
||||
|
|
||||
|
keyBase64 := base64.StdEncoding.EncodeToString(key) |
||||
|
md5sum2 := md5.Sum(key) |
||||
|
keyMD5 := base64.StdEncoding.EncodeToString(md5sum2[:]) |
||||
|
|
||||
|
req.Header.Set(s3_constants.AmzCopySourceServerSideEncryptionCustomerAlgorithm, "AES256") |
||||
|
req.Header.Set(s3_constants.AmzCopySourceServerSideEncryptionCustomerKey, keyBase64) |
||||
|
req.Header.Set(s3_constants.AmzCopySourceServerSideEncryptionCustomerKeyMD5, keyMD5) |
||||
|
|
||||
|
// Test parsing copy source headers
|
||||
|
customerKey, err := ParseSSECCopySourceHeaders(req) |
||||
|
if err != nil { |
||||
|
t.Errorf("Expected successful copy source parsing, got error: %v", err) |
||||
|
} |
||||
|
|
||||
|
if customerKey == nil { |
||||
|
t.Error("Expected customer key from copy source headers, got nil") |
||||
|
} |
||||
|
|
||||
|
if customerKey.Algorithm != "AES256" { |
||||
|
t.Errorf("Expected algorithm AES256, got %s", customerKey.Algorithm) |
||||
|
} |
||||
|
|
||||
|
if !bytes.Equal(customerKey.Key, key) { |
||||
|
t.Error("Copy source key doesn't match original") |
||||
|
} |
||||
|
|
||||
|
// Test that regular headers don't interfere with copy source headers
|
||||
|
regularKey, err := ParseSSECHeaders(req) |
||||
|
if err != nil { |
||||
|
t.Errorf("Regular header parsing should not fail: %v", err) |
||||
|
} |
||||
|
|
||||
|
if regularKey != nil { |
||||
|
t.Error("Expected nil for regular headers when only copy source headers are present") |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestSSECHeaderValidationErrors(t *testing.T) { |
||||
|
tests := []struct { |
||||
|
name string |
||||
|
algorithm string |
||||
|
key string |
||||
|
keyMD5 string |
||||
|
wantErr error |
||||
|
}{ |
||||
|
{ |
||||
|
name: "invalid algorithm", |
||||
|
algorithm: "AES128", |
||||
|
key: base64.StdEncoding.EncodeToString(make([]byte, 32)), |
||||
|
keyMD5: base64MD5(make([]byte, 32)), |
||||
|
wantErr: ErrInvalidEncryptionAlgorithm, |
||||
|
}, |
||||
|
{ |
||||
|
name: "invalid key length", |
||||
|
algorithm: "AES256", |
||||
|
key: base64.StdEncoding.EncodeToString(make([]byte, 16)), |
||||
|
keyMD5: base64MD5(make([]byte, 16)), |
||||
|
wantErr: ErrInvalidEncryptionKey, |
||||
|
}, |
||||
|
{ |
||||
|
name: "mismatched MD5", |
||||
|
algorithm: "AES256", |
||||
|
key: base64.StdEncoding.EncodeToString(make([]byte, 32)), |
||||
|
keyMD5: "wrong==md5", |
||||
|
wantErr: ErrSSECustomerKeyMD5Mismatch, |
||||
|
}, |
||||
|
{ |
||||
|
name: "incomplete headers", |
||||
|
algorithm: "AES256", |
||||
|
key: "", |
||||
|
keyMD5: "", |
||||
|
wantErr: ErrInvalidRequest, |
||||
|
}, |
||||
|
} |
||||
|
|
||||
|
for _, tt := range tests { |
||||
|
t.Run(tt.name, func(t *testing.T) { |
||||
|
req := &http.Request{Header: make(http.Header)} |
||||
|
|
||||
|
if tt.algorithm != "" { |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerAlgorithm, tt.algorithm) |
||||
|
} |
||||
|
if tt.key != "" { |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKey, tt.key) |
||||
|
} |
||||
|
if tt.keyMD5 != "" { |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerKeyMD5, tt.keyMD5) |
||||
|
} |
||||
|
|
||||
|
err := ValidateSSECHeaders(req) |
||||
|
if err != tt.wantErr { |
||||
|
t.Errorf("Expected error %v, got %v", tt.wantErr, err) |
||||
|
} |
||||
|
}) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestSSECEncryptionDecryption(t *testing.T) { |
||||
|
// Create customer key
|
||||
|
key := make([]byte, 32) |
||||
|
for i := range key { |
||||
|
key[i] = byte(i) |
||||
|
} |
||||
|
|
||||
|
md5sumKey := md5.Sum(key) |
||||
|
customerKey := &SSECustomerKey{ |
||||
|
Algorithm: "AES256", |
||||
|
Key: key, |
||||
|
KeyMD5: base64.StdEncoding.EncodeToString(md5sumKey[:]), |
||||
|
} |
||||
|
|
||||
|
// Test data
|
||||
|
testData := []byte("Hello, World! This is a test of SSE-C encryption.") |
||||
|
|
||||
|
// Create encrypted reader
|
||||
|
dataReader := bytes.NewReader(testData) |
||||
|
encryptedReader, err := CreateSSECEncryptedReader(dataReader, customerKey) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create encrypted reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Read encrypted data
|
||||
|
encryptedData, err := io.ReadAll(encryptedReader) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to read encrypted data: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Verify data is actually encrypted (different from original)
|
||||
|
if bytes.Equal(encryptedData[16:], testData) { // Skip IV
|
||||
|
t.Error("Data doesn't appear to be encrypted") |
||||
|
} |
||||
|
|
||||
|
// Create decrypted reader
|
||||
|
encryptedReader2 := bytes.NewReader(encryptedData) |
||||
|
decryptedReader, err := CreateSSECDecryptedReader(encryptedReader2, customerKey) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create decrypted reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Read decrypted data
|
||||
|
decryptedData, err := io.ReadAll(decryptedReader) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to read decrypted data: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Verify decrypted data matches original
|
||||
|
if !bytes.Equal(decryptedData, testData) { |
||||
|
t.Errorf("Decrypted data doesn't match original.\nOriginal: %s\nDecrypted: %s", testData, decryptedData) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestSSECIsSSECRequest(t *testing.T) { |
||||
|
// Test with SSE-C headers
|
||||
|
req := &http.Request{Header: make(http.Header)} |
||||
|
req.Header.Set(s3_constants.AmzServerSideEncryptionCustomerAlgorithm, "AES256") |
||||
|
|
||||
|
if !IsSSECRequest(req) { |
||||
|
t.Error("Expected IsSSECRequest to return true when SSE-C headers are present") |
||||
|
} |
||||
|
|
||||
|
// Test without SSE-C headers
|
||||
|
req2 := &http.Request{Header: make(http.Header)} |
||||
|
if IsSSECRequest(req2) { |
||||
|
t.Error("Expected IsSSECRequest to return false when no SSE-C headers are present") |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Test encryption with different data sizes (similar to s3tests)
|
||||
|
func TestSSECEncryptionVariousSizes(t *testing.T) { |
||||
|
sizes := []int{1, 13, 1024, 1024 * 1024} // 1B, 13B, 1KB, 1MB
|
||||
|
|
||||
|
for _, size := range sizes { |
||||
|
t.Run(fmt.Sprintf("size_%d", size), func(t *testing.T) { |
||||
|
// Create customer key
|
||||
|
key := make([]byte, 32) |
||||
|
for i := range key { |
||||
|
key[i] = byte(i + size) // Make key unique per test
|
||||
|
} |
||||
|
|
||||
|
md5sumDyn := md5.Sum(key) |
||||
|
customerKey := &SSECustomerKey{ |
||||
|
Algorithm: "AES256", |
||||
|
Key: key, |
||||
|
KeyMD5: base64.StdEncoding.EncodeToString(md5sumDyn[:]), |
||||
|
} |
||||
|
|
||||
|
// Create test data of specified size
|
||||
|
testData := make([]byte, size) |
||||
|
for i := range testData { |
||||
|
testData[i] = byte('A' + (i % 26)) // Pattern of A-Z
|
||||
|
} |
||||
|
|
||||
|
// Encrypt
|
||||
|
dataReader := bytes.NewReader(testData) |
||||
|
encryptedReader, err := CreateSSECEncryptedReader(dataReader, customerKey) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create encrypted reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
encryptedData, err := io.ReadAll(encryptedReader) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to read encrypted data: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Verify IV is present and data is encrypted
|
||||
|
if len(encryptedData) < AESBlockSize { |
||||
|
t.Fatalf("Encrypted data too short, missing IV") |
||||
|
} |
||||
|
|
||||
|
if len(encryptedData) != size+AESBlockSize { |
||||
|
t.Errorf("Expected encrypted data length %d, got %d", size+AESBlockSize, len(encryptedData)) |
||||
|
} |
||||
|
|
||||
|
// Decrypt
|
||||
|
encryptedReader2 := bytes.NewReader(encryptedData) |
||||
|
decryptedReader, err := CreateSSECDecryptedReader(encryptedReader2, customerKey) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create decrypted reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
decryptedData, err := io.ReadAll(decryptedReader) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to read decrypted data: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Verify decrypted data matches original
|
||||
|
if !bytes.Equal(decryptedData, testData) { |
||||
|
t.Errorf("Decrypted data doesn't match original for size %d", size) |
||||
|
} |
||||
|
}) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestSSECEncryptionWithNilKey(t *testing.T) { |
||||
|
testData := []byte("test data") |
||||
|
dataReader := bytes.NewReader(testData) |
||||
|
|
||||
|
// Test encryption with nil key (should pass through)
|
||||
|
encryptedReader, err := CreateSSECEncryptedReader(dataReader, nil) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create encrypted reader with nil key: %v", err) |
||||
|
} |
||||
|
|
||||
|
result, err := io.ReadAll(encryptedReader) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to read from pass-through reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
if !bytes.Equal(result, testData) { |
||||
|
t.Error("Data should pass through unchanged when key is nil") |
||||
|
} |
||||
|
|
||||
|
// Test decryption with nil key (should pass through)
|
||||
|
dataReader2 := bytes.NewReader(testData) |
||||
|
decryptedReader, err := CreateSSECDecryptedReader(dataReader2, nil) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create decrypted reader with nil key: %v", err) |
||||
|
} |
||||
|
|
||||
|
result2, err := io.ReadAll(decryptedReader) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to read from pass-through reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
if !bytes.Equal(result2, testData) { |
||||
|
t.Error("Data should pass through unchanged when key is nil") |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// TestSSECEncryptionSmallBuffers tests the fix for the critical bug where small buffers
|
||||
|
// could corrupt the data stream when reading in chunks smaller than the IV size
|
||||
|
func TestSSECEncryptionSmallBuffers(t *testing.T) { |
||||
|
testData := []byte("This is a test message for small buffer reads") |
||||
|
|
||||
|
// Create customer key
|
||||
|
key := make([]byte, 32) |
||||
|
for i := range key { |
||||
|
key[i] = byte(i) |
||||
|
} |
||||
|
|
||||
|
md5sumKey3 := md5.Sum(key) |
||||
|
customerKey := &SSECustomerKey{ |
||||
|
Algorithm: "AES256", |
||||
|
Key: key, |
||||
|
KeyMD5: base64.StdEncoding.EncodeToString(md5sumKey3[:]), |
||||
|
} |
||||
|
|
||||
|
// Create encrypted reader
|
||||
|
dataReader := bytes.NewReader(testData) |
||||
|
encryptedReader, err := CreateSSECEncryptedReader(dataReader, customerKey) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create encrypted reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Read with very small buffers (smaller than IV size of 16 bytes)
|
||||
|
var encryptedData []byte |
||||
|
smallBuffer := make([]byte, 5) // Much smaller than 16-byte IV
|
||||
|
|
||||
|
for { |
||||
|
n, err := encryptedReader.Read(smallBuffer) |
||||
|
if n > 0 { |
||||
|
encryptedData = append(encryptedData, smallBuffer[:n]...) |
||||
|
} |
||||
|
if err == io.EOF { |
||||
|
break |
||||
|
} |
||||
|
if err != nil { |
||||
|
t.Fatalf("Error reading encrypted data: %v", err) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Verify the encrypted data starts with 16-byte IV
|
||||
|
if len(encryptedData) < 16 { |
||||
|
t.Fatalf("Encrypted data too short, expected at least 16 bytes for IV, got %d", len(encryptedData)) |
||||
|
} |
||||
|
|
||||
|
// Expected total size: 16 bytes (IV) + len(testData)
|
||||
|
expectedSize := 16 + len(testData) |
||||
|
if len(encryptedData) != expectedSize { |
||||
|
t.Errorf("Expected encrypted data size %d, got %d", expectedSize, len(encryptedData)) |
||||
|
} |
||||
|
|
||||
|
// Decrypt and verify
|
||||
|
encryptedReader2 := bytes.NewReader(encryptedData) |
||||
|
decryptedReader, err := CreateSSECDecryptedReader(encryptedReader2, customerKey) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to create decrypted reader: %v", err) |
||||
|
} |
||||
|
|
||||
|
decryptedData, err := io.ReadAll(decryptedReader) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to read decrypted data: %v", err) |
||||
|
} |
||||
|
|
||||
|
if !bytes.Equal(decryptedData, testData) { |
||||
|
t.Errorf("Decrypted data doesn't match original.\nOriginal: %s\nDecrypted: %s", testData, decryptedData) |
||||
|
} |
||||
|
} |
Write
Preview
Loading…
Cancel
Save
Reference in new issue