You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

209 lines
7.0 KiB

6 years ago
6 years ago
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
3 years ago
4 years ago
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
3 years ago
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158
3 years ago
5 years ago
3 years ago
3 years ago
4 years ago
  1. package weed_server
  2. import (
  3. "context"
  4. "fmt"
  5. "net/http"
  6. "os"
  7. "sync"
  8. "time"
  9. "github.com/seaweedfs/seaweedfs/weed/stats"
  10. "google.golang.org/grpc"
  11. "github.com/seaweedfs/seaweedfs/weed/util/grace"
  12. "github.com/seaweedfs/seaweedfs/weed/operation"
  13. "github.com/seaweedfs/seaweedfs/weed/pb"
  14. "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
  15. "github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
  16. "github.com/seaweedfs/seaweedfs/weed/util"
  17. "github.com/seaweedfs/seaweedfs/weed/filer"
  18. _ "github.com/seaweedfs/seaweedfs/weed/filer/arangodb"
  19. _ "github.com/seaweedfs/seaweedfs/weed/filer/cassandra"
  20. _ "github.com/seaweedfs/seaweedfs/weed/filer/elastic/v7"
  21. _ "github.com/seaweedfs/seaweedfs/weed/filer/etcd"
  22. _ "github.com/seaweedfs/seaweedfs/weed/filer/hbase"
  23. _ "github.com/seaweedfs/seaweedfs/weed/filer/leveldb"
  24. _ "github.com/seaweedfs/seaweedfs/weed/filer/leveldb2"
  25. _ "github.com/seaweedfs/seaweedfs/weed/filer/leveldb3"
  26. _ "github.com/seaweedfs/seaweedfs/weed/filer/mongodb"
  27. _ "github.com/seaweedfs/seaweedfs/weed/filer/mysql"
  28. _ "github.com/seaweedfs/seaweedfs/weed/filer/mysql2"
  29. _ "github.com/seaweedfs/seaweedfs/weed/filer/postgres"
  30. _ "github.com/seaweedfs/seaweedfs/weed/filer/postgres2"
  31. _ "github.com/seaweedfs/seaweedfs/weed/filer/redis"
  32. _ "github.com/seaweedfs/seaweedfs/weed/filer/redis2"
  33. _ "github.com/seaweedfs/seaweedfs/weed/filer/redis3"
  34. _ "github.com/seaweedfs/seaweedfs/weed/filer/sqlite"
  35. _ "github.com/seaweedfs/seaweedfs/weed/filer/ydb"
  36. "github.com/seaweedfs/seaweedfs/weed/glog"
  37. "github.com/seaweedfs/seaweedfs/weed/notification"
  38. _ "github.com/seaweedfs/seaweedfs/weed/notification/aws_sqs"
  39. _ "github.com/seaweedfs/seaweedfs/weed/notification/gocdk_pub_sub"
  40. _ "github.com/seaweedfs/seaweedfs/weed/notification/google_pub_sub"
  41. _ "github.com/seaweedfs/seaweedfs/weed/notification/kafka"
  42. _ "github.com/seaweedfs/seaweedfs/weed/notification/log"
  43. "github.com/seaweedfs/seaweedfs/weed/security"
  44. )
  45. type FilerOption struct {
  46. Masters map[string]pb.ServerAddress
  47. FilerGroup string
  48. Collection string
  49. DefaultReplication string
  50. DisableDirListing bool
  51. MaxMB int
  52. DirListingLimit int
  53. DataCenter string
  54. Rack string
  55. DataNode string
  56. DefaultLevelDbDir string
  57. DisableHttp bool
  58. Host pb.ServerAddress
  59. recursiveDelete bool
  60. Cipher bool
  61. SaveToFilerLimit int64
  62. ConcurrentUploadLimit int64
  63. ShowUIDirectoryDelete bool
  64. }
  65. type FilerServer struct {
  66. filer_pb.UnimplementedSeaweedFilerServer
  67. option *FilerOption
  68. secret security.SigningKey
  69. filer *filer.Filer
  70. filerGuard *security.Guard
  71. grpcDialOption grpc.DialOption
  72. // metrics read from the master
  73. metricsAddress string
  74. metricsIntervalSec int
  75. // notifying clients
  76. listenersLock sync.Mutex
  77. listenersCond *sync.Cond
  78. // track known metadata listeners
  79. knownListenersLock sync.Mutex
  80. knownListeners map[int32]int32
  81. inFlightDataSize int64
  82. inFlightDataLimitCond *sync.Cond
  83. }
  84. func NewFilerServer(defaultMux, readonlyMux *http.ServeMux, option *FilerOption) (fs *FilerServer, err error) {
  85. v := util.GetViper()
  86. signingKey := v.GetString("jwt.filer_signing.key")
  87. v.SetDefault("jwt.filer_signing.expires_after_seconds", 10)
  88. expiresAfterSec := v.GetInt("jwt.filer_signing.expires_after_seconds")
  89. readSigningKey := v.GetString("jwt.filer_signing.read.key")
  90. v.SetDefault("jwt.filer_signing.read.expires_after_seconds", 60)
  91. readExpiresAfterSec := v.GetInt("jwt.filer_signing.read.expires_after_seconds")
  92. fs = &FilerServer{
  93. option: option,
  94. grpcDialOption: security.LoadClientTLS(util.GetViper(), "grpc.filer"),
  95. knownListeners: make(map[int32]int32),
  96. inFlightDataLimitCond: sync.NewCond(new(sync.Mutex)),
  97. }
  98. fs.listenersCond = sync.NewCond(&fs.listenersLock)
  99. if len(option.Masters) == 0 {
  100. glog.Fatal("master list is required!")
  101. }
  102. fs.filer = filer.NewFiler(option.Masters, fs.grpcDialOption, option.Host, option.FilerGroup, option.Collection, option.DefaultReplication, option.DataCenter, func() {
  103. fs.listenersCond.Broadcast()
  104. })
  105. fs.filer.Cipher = option.Cipher
  106. // we do not support IP whitelist right now
  107. fs.filerGuard = security.NewGuard([]string{}, signingKey, expiresAfterSec, readSigningKey, readExpiresAfterSec)
  108. fs.checkWithMaster()
  109. go stats.LoopPushingMetric("filer", string(fs.option.Host), fs.metricsAddress, fs.metricsIntervalSec)
  110. go fs.filer.KeepMasterClientConnected()
  111. if !util.LoadConfiguration("filer", false) {
  112. v.SetDefault("leveldb2.enabled", true)
  113. v.SetDefault("leveldb2.dir", option.DefaultLevelDbDir)
  114. _, err := os.Stat(option.DefaultLevelDbDir)
  115. if os.IsNotExist(err) {
  116. os.MkdirAll(option.DefaultLevelDbDir, 0755)
  117. }
  118. glog.V(0).Infof("default to create filer store dir in %s", option.DefaultLevelDbDir)
  119. } else {
  120. glog.Warningf("skipping default store dir in %s", option.DefaultLevelDbDir)
  121. }
  122. util.LoadConfiguration("notification", false)
  123. fs.option.recursiveDelete = v.GetBool("filer.options.recursive_delete")
  124. v.SetDefault("filer.options.buckets_folder", "/buckets")
  125. fs.filer.DirBucketsPath = v.GetString("filer.options.buckets_folder")
  126. // TODO deprecated, will be be removed after 2020-12-31
  127. // replaced by https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration
  128. // fs.filer.FsyncBuckets = v.GetStringSlice("filer.options.buckets_fsync")
  129. isFresh := fs.filer.LoadConfiguration(v)
  130. notification.LoadConfiguration(v, "notification.")
  131. handleStaticResources(defaultMux)
  132. if !option.DisableHttp {
  133. defaultMux.HandleFunc("/", fs.filerHandler)
  134. }
  135. if defaultMux != readonlyMux {
  136. handleStaticResources(readonlyMux)
  137. readonlyMux.HandleFunc("/", fs.readonlyFilerHandler)
  138. }
  139. existingNodes := fs.filer.ListExistingPeerUpdates()
  140. startFromTime := time.Now().Add(-filer.LogFlushInterval)
  141. if isFresh {
  142. glog.V(0).Infof("%s bootstrap from peers %+v", option.Host, existingNodes)
  143. if err := fs.filer.MaybeBootstrapFromPeers(option.Host, existingNodes, startFromTime); err != nil {
  144. glog.Fatalf("%s bootstrap from %+v", option.Host, existingNodes)
  145. }
  146. }
  147. fs.filer.AggregateFromPeers(option.Host, existingNodes, startFromTime)
  148. fs.filer.LoadFilerConf()
  149. fs.filer.LoadRemoteStorageConfAndMapping()
  150. grace.OnInterrupt(func() {
  151. fs.filer.Shutdown()
  152. })
  153. return fs, nil
  154. }
  155. func (fs *FilerServer) checkWithMaster() {
  156. isConnected := false
  157. for !isConnected {
  158. for _, master := range fs.option.Masters {
  159. readErr := operation.WithMasterServerClient(false, master, fs.grpcDialOption, func(masterClient master_pb.SeaweedClient) error {
  160. resp, err := masterClient.GetMasterConfiguration(context.Background(), &master_pb.GetMasterConfigurationRequest{})
  161. if err != nil {
  162. return fmt.Errorf("get master %s configuration: %v", master, err)
  163. }
  164. fs.metricsAddress, fs.metricsIntervalSec = resp.MetricsAddress, int(resp.MetricsIntervalSeconds)
  165. if fs.option.DefaultReplication == "" {
  166. fs.option.DefaultReplication = resp.DefaultReplication
  167. }
  168. return nil
  169. })
  170. if readErr == nil {
  171. isConnected = true
  172. } else {
  173. time.Sleep(7 * time.Second)
  174. }
  175. }
  176. }
  177. }