You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

223 lines
7.7 KiB

6 years ago
6 years ago
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
3 years ago
4 years ago
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
3 years ago
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
3 years ago
5 years ago
3 years ago
3 years ago
2 years ago
  1. package weed_server
  2. import (
  3. "context"
  4. "fmt"
  5. "net/http"
  6. "os"
  7. "strings"
  8. "sync"
  9. "time"
  10. "github.com/seaweedfs/seaweedfs/weed/stats"
  11. "google.golang.org/grpc"
  12. "github.com/seaweedfs/seaweedfs/weed/util/grace"
  13. "github.com/seaweedfs/seaweedfs/weed/operation"
  14. "github.com/seaweedfs/seaweedfs/weed/pb"
  15. "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
  16. "github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
  17. "github.com/seaweedfs/seaweedfs/weed/util"
  18. "github.com/seaweedfs/seaweedfs/weed/filer"
  19. _ "github.com/seaweedfs/seaweedfs/weed/filer/arangodb"
  20. _ "github.com/seaweedfs/seaweedfs/weed/filer/cassandra"
  21. _ "github.com/seaweedfs/seaweedfs/weed/filer/elastic/v7"
  22. _ "github.com/seaweedfs/seaweedfs/weed/filer/etcd"
  23. _ "github.com/seaweedfs/seaweedfs/weed/filer/hbase"
  24. _ "github.com/seaweedfs/seaweedfs/weed/filer/leveldb"
  25. _ "github.com/seaweedfs/seaweedfs/weed/filer/leveldb2"
  26. _ "github.com/seaweedfs/seaweedfs/weed/filer/leveldb3"
  27. _ "github.com/seaweedfs/seaweedfs/weed/filer/mongodb"
  28. _ "github.com/seaweedfs/seaweedfs/weed/filer/mysql"
  29. _ "github.com/seaweedfs/seaweedfs/weed/filer/mysql2"
  30. _ "github.com/seaweedfs/seaweedfs/weed/filer/postgres"
  31. _ "github.com/seaweedfs/seaweedfs/weed/filer/postgres2"
  32. _ "github.com/seaweedfs/seaweedfs/weed/filer/redis"
  33. _ "github.com/seaweedfs/seaweedfs/weed/filer/redis2"
  34. _ "github.com/seaweedfs/seaweedfs/weed/filer/redis3"
  35. _ "github.com/seaweedfs/seaweedfs/weed/filer/sqlite"
  36. _ "github.com/seaweedfs/seaweedfs/weed/filer/ydb"
  37. "github.com/seaweedfs/seaweedfs/weed/glog"
  38. "github.com/seaweedfs/seaweedfs/weed/notification"
  39. _ "github.com/seaweedfs/seaweedfs/weed/notification/aws_sqs"
  40. _ "github.com/seaweedfs/seaweedfs/weed/notification/gocdk_pub_sub"
  41. _ "github.com/seaweedfs/seaweedfs/weed/notification/google_pub_sub"
  42. _ "github.com/seaweedfs/seaweedfs/weed/notification/kafka"
  43. _ "github.com/seaweedfs/seaweedfs/weed/notification/log"
  44. "github.com/seaweedfs/seaweedfs/weed/security"
  45. )
  46. type FilerOption struct {
  47. Masters *pb.ServerDiscovery
  48. FilerGroup string
  49. Collection string
  50. DefaultReplication string
  51. DisableDirListing bool
  52. MaxMB int
  53. DirListingLimit int
  54. DataCenter string
  55. Rack string
  56. DataNode string
  57. DefaultLevelDbDir string
  58. DisableHttp bool
  59. Host pb.ServerAddress
  60. recursiveDelete bool
  61. Cipher bool
  62. SaveToFilerLimit int64
  63. ConcurrentUploadLimit int64
  64. ShowUIDirectoryDelete bool
  65. DownloadMaxBytesPs int64
  66. DiskType string
  67. AllowedOrigins []string
  68. }
  69. type FilerServer struct {
  70. inFlightDataSize int64
  71. inFlightDataLimitCond *sync.Cond
  72. filer_pb.UnimplementedSeaweedFilerServer
  73. option *FilerOption
  74. secret security.SigningKey
  75. filer *filer.Filer
  76. filerGuard *security.Guard
  77. grpcDialOption grpc.DialOption
  78. // metrics read from the master
  79. metricsAddress string
  80. metricsIntervalSec int
  81. // notifying clients
  82. listenersLock sync.Mutex
  83. listenersCond *sync.Cond
  84. // track known metadata listeners
  85. knownListenersLock sync.Mutex
  86. knownListeners map[int32]int32
  87. }
  88. func NewFilerServer(defaultMux, readonlyMux *http.ServeMux, option *FilerOption) (fs *FilerServer, err error) {
  89. v := util.GetViper()
  90. signingKey := v.GetString("jwt.filer_signing.key")
  91. v.SetDefault("jwt.filer_signing.expires_after_seconds", 10)
  92. expiresAfterSec := v.GetInt("jwt.filer_signing.expires_after_seconds")
  93. readSigningKey := v.GetString("jwt.filer_signing.read.key")
  94. v.SetDefault("jwt.filer_signing.read.expires_after_seconds", 60)
  95. readExpiresAfterSec := v.GetInt("jwt.filer_signing.read.expires_after_seconds")
  96. v.SetDefault("cors.allowed_origins.values", "*")
  97. allowedOrigins := v.GetString("cors.allowed_origins.values")
  98. domains := strings.Split(allowedOrigins, ",")
  99. option.AllowedOrigins = domains
  100. fs = &FilerServer{
  101. option: option,
  102. grpcDialOption: security.LoadClientTLS(util.GetViper(), "grpc.filer"),
  103. knownListeners: make(map[int32]int32),
  104. inFlightDataLimitCond: sync.NewCond(new(sync.Mutex)),
  105. }
  106. fs.listenersCond = sync.NewCond(&fs.listenersLock)
  107. option.Masters.RefreshBySrvIfAvailable()
  108. if len(option.Masters.GetInstances()) == 0 {
  109. glog.Fatal("master list is required!")
  110. }
  111. v.SetDefault("filer.options.max_file_name_length", 255)
  112. maxFilenameLength := v.GetUint32("filer.options.max_file_name_length")
  113. fs.filer = filer.NewFiler(*option.Masters, fs.grpcDialOption, option.Host, option.FilerGroup, option.Collection, option.DefaultReplication, option.DataCenter, maxFilenameLength, func() {
  114. fs.listenersCond.Broadcast()
  115. })
  116. fs.filer.Cipher = option.Cipher
  117. // we do not support IP whitelist right now
  118. fs.filerGuard = security.NewGuard([]string{}, signingKey, expiresAfterSec, readSigningKey, readExpiresAfterSec)
  119. fs.checkWithMaster()
  120. go stats.LoopPushingMetric("filer", string(fs.option.Host), fs.metricsAddress, fs.metricsIntervalSec)
  121. go fs.filer.KeepMasterClientConnected()
  122. if !util.LoadConfiguration("filer", false) {
  123. v.SetDefault("leveldb2.enabled", true)
  124. v.SetDefault("leveldb2.dir", option.DefaultLevelDbDir)
  125. _, err := os.Stat(option.DefaultLevelDbDir)
  126. if os.IsNotExist(err) {
  127. os.MkdirAll(option.DefaultLevelDbDir, 0755)
  128. }
  129. glog.V(0).Infof("default to create filer store dir in %s", option.DefaultLevelDbDir)
  130. } else {
  131. glog.Warningf("skipping default store dir in %s", option.DefaultLevelDbDir)
  132. }
  133. util.LoadConfiguration("notification", false)
  134. fs.option.recursiveDelete = v.GetBool("filer.options.recursive_delete")
  135. v.SetDefault("filer.options.buckets_folder", "/buckets")
  136. fs.filer.DirBucketsPath = v.GetString("filer.options.buckets_folder")
  137. // TODO deprecated, will be be removed after 2020-12-31
  138. // replaced by https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration
  139. // fs.filer.FsyncBuckets = v.GetStringSlice("filer.options.buckets_fsync")
  140. isFresh := fs.filer.LoadConfiguration(v)
  141. notification.LoadConfiguration(v, "notification.")
  142. handleStaticResources(defaultMux)
  143. if !option.DisableHttp {
  144. defaultMux.HandleFunc("/healthz", fs.filerHealthzHandler)
  145. defaultMux.HandleFunc("/", fs.filerHandler)
  146. }
  147. if defaultMux != readonlyMux {
  148. handleStaticResources(readonlyMux)
  149. readonlyMux.HandleFunc("/healthz", fs.filerHealthzHandler)
  150. readonlyMux.HandleFunc("/", fs.readonlyFilerHandler)
  151. }
  152. existingNodes := fs.filer.ListExistingPeerUpdates()
  153. startFromTime := time.Now().Add(-filer.LogFlushInterval)
  154. if isFresh {
  155. glog.V(0).Infof("%s bootstrap from peers %+v", option.Host, existingNodes)
  156. if err := fs.filer.MaybeBootstrapFromPeers(option.Host, existingNodes, startFromTime); err != nil {
  157. glog.Fatalf("%s bootstrap from %+v: %v", option.Host, existingNodes, err)
  158. }
  159. }
  160. fs.filer.AggregateFromPeers(option.Host, existingNodes, startFromTime)
  161. fs.filer.LoadFilerConf()
  162. fs.filer.LoadRemoteStorageConfAndMapping()
  163. grace.OnInterrupt(func() {
  164. fs.filer.Shutdown()
  165. })
  166. fs.filer.Dlm.LockRing.SetTakeSnapshotCallback(fs.OnDlmChangeSnapshot)
  167. return fs, nil
  168. }
  169. func (fs *FilerServer) checkWithMaster() {
  170. isConnected := false
  171. for !isConnected {
  172. fs.option.Masters.RefreshBySrvIfAvailable()
  173. for _, master := range fs.option.Masters.GetInstances() {
  174. readErr := operation.WithMasterServerClient(false, master, fs.grpcDialOption, func(masterClient master_pb.SeaweedClient) error {
  175. resp, err := masterClient.GetMasterConfiguration(context.Background(), &master_pb.GetMasterConfigurationRequest{})
  176. if err != nil {
  177. return fmt.Errorf("get master %s configuration: %v", master, err)
  178. }
  179. fs.metricsAddress, fs.metricsIntervalSec = resp.MetricsAddress, int(resp.MetricsIntervalSeconds)
  180. return nil
  181. })
  182. if readErr == nil {
  183. isConnected = true
  184. } else {
  185. time.Sleep(7 * time.Second)
  186. }
  187. }
  188. }
  189. }