数据流设置控制器服务账户

Question

我尝试为Dataflow设置控制器服务账户。在我的dataflow选项中，我有：

options.setGcpCredential(GoogleCredentials.fromStream(
                         new FileInputStream("key.json")).createScoped(someArrays)); 
options.setServiceAccount("[e-mail]");

但我得到的是：

WARNING: Request failed with code 403, performed 0 retries due to IOExceptions,         
         performed 0 retries due to unsuccessful status codes, HTTP framework says 
         request can be retried, (caller responsible for retrying): 
         https://dataflow.googleapis.com/v1b3/projects/MYPROJECT/locations/MYLOCATION/jobs
Exception in thread "main" java.lang.RuntimeException: Failed to create a workflow 
         job: (CODE): Current user cannot act as 
         service account "[e-mail]. 
         Causes: (CODE): Current user cannot act as 
         service account "[e-mail].
    at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:791)
    at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:173)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)

抱歉，我只能回答英文问题。

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "(CODE): Current user cannot act as service account 
                 [e-mail]. Causes: (CODE): Current user
                 cannot act as service account [e-mail].",
    "reason" : "forbidden"
  } ],
  "message" : "(CODE): Current user cannot act as service account 
               [e-mail]. Causes: (CODE): Current user 
               cannot act as service account [e-mail].",
  "status" : "PERMISSION_DENIED"
}

我是否缺少某些角色或权限？

- Magda Kiwi

Answer 1

也许有人会发现这很有帮助：

• 对于控制器而言，它是：数据流工作人员和存储对象管理员（在Google文档中找到）。

• 对于执行器而言，它是：服务账号用户。

- Magda Kiwi

---

对于 Terraform，我使用了： roles = ["dataflow.admin", "dataflow.worker", "storage.admin", "iam.serviceAccountUser"] - Kush

@Kush - 谢谢，非常感谢。这解决了我的问题。 - abanmitra

注意：基于最小特权原则，您可能希望将角色限制在各自的资源上（例如，仅适用于存储桶而不是项目级别，还应在特定服务帐号级别上授予服务帐号用户，而不是在项目级别上授予）。 - Pinguin Dirk

如果您只是为了部署Dataflow作业而使用服务帐户，我认为dataflow.worker角色是不必要的？ - André Laszlo

Answer 2

我一直遇到这个错误，认为分享我的经验很有价值（部分原因是我怀疑将来会再次遇到这个问题）。

创建数据流作业的Terraform代码如下：

resource "google_dataflow_job" "wordcount" {
  # https://stackoverflow.com/a/59931467/201657
  name              = "wordcount"
  template_gcs_path = "gs://dataflow-templates/latest/Word_Count"
  temp_gcs_location = "gs://${local.name-prefix}-functions/temp"
  parameters = {
    inputFile = "gs://dataflow-samples/shakespeare/kinglear.txt"
    output = "gs://${local.name-prefix}-functions/wordcount/output"
  }
  service_account_email = "serviceAccount:${data.google_service_account.sa.email}"
}

错误信息:

错误: googleapi: 错误 400: (c3c0d991927a8658): 当前用户不能作为服务帐户 serviceAccount:[e-mail]. 进行操作, badRequest

运行terraform apply时返回了该错误消息。查看提供的日志可以获得更多信息：

gcloud logging read 'timestamp >= "2020-12-31T13:39:58.733249492Z" AND timestamp <= "2020-12-31T13:45:58.733249492Z"' --format="csv(timestamp,severity,textPayload)" --order=asc

返回了各种日志记录，包括以下内容：

控制器服务帐户的权限验证失败。IAM角色roles/dataflow.worker应授予控制器服务帐户[e-mail]。

因此，我授予了缺少的角色授权。

gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:dataflowdemo@${PROJECT}.iam.gserviceaccount.com" \
  --role="roles/dataflow.worker"

然后我再次运行了 terraform apply 命令。这一次，虽然在 terraform 输出中出现了相同的错误，但是日志中没有看到任何错误。

接着，我按照https://cloud.google.com/dataflow/docs/concepts/access-control#creating_jobs 上给出的建议，授予了 roles/dataflow.admin 的权限。

gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:dataflowdemo@${PROJECT}.iam.gserviceaccount.com" \
  --role="roles/dataflow.admin"

但是和之前的尝试相比，并没有明显的区别。

然后我尝试打开terraform调试日志，得到了如下信息：

2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: ---[ REQUEST ]---------------------------------------
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: POST /v1b3/projects/redacted/locations/europe-west1/templates?alt=json&prettyPrint=false HTTP/1.1
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Host: dataflow.googleapis.com
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: User-Agent: google-api-go-client/0.5 Terraform/0.14.2 (+https://www.terraform.io) Terraform-Plugin-SDK/2.1.0 terraform-provider-google/dev
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Content-Length: 385
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Content-Type: application/json
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: X-Goog-Api-Client: gl-go/1.14.5 gdcl/20201023
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Accept-Encoding: gzip
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: {
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:  "environment": {
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:   "serviceAccountEmail": "serviceAccount:[e-mail]",
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:   "tempLocation": "gs://jamiet-demo-functions/temp"
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:  },
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:  "gcsPath": "gs://dataflow-templates/latest/Word_Count",
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:  "jobName": "wordcount",
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:  "parameters": {
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:   "inputFile": "gs://dataflow-samples/shakespeare/kinglear.txt",
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:   "output": "gs://jamiet-demo-functions/wordcount/output"
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:  }
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: }
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:
2020-12-31T16:04:13.129Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: -----------------------------------------------------
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: 2020/12/31 16:04:14 [DEBUG] Google API Response Details:
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: ---[ RESPONSE ]--------------------------------------
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: HTTP/1.1 400 Bad Request
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Connection: close
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Transfer-Encoding: chunked
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Cache-Control: private
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Content-Type: application/json; charset=UTF-8
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Date: Thu, 31 Dec 2020 16:04:15 GMT
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Server: ESF
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Vary: Origin
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Vary: X-Origin
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: Vary: Referer
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: X-Content-Type-Options: nosniff
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: X-Frame-Options: SAMEORIGIN
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: X-Xss-Protection: 0
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: 1f9
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: {
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:   "error": {
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:     "code": 400,
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:     "message": "(dbacb1c39beb28c9): Current user cannot act as service account serviceAccount:[e-mail].",
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:     "errors": [
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:       {
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:         "message": "(dbacb1c39beb28c9): Current user cannot act as service account serviceAccount:[e-mail].",
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:         "domain": "global",
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:         "reason": "badRequest"
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:       }
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:     ],
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:     "status": "INVALID_ARGUMENT"
orm-provider-google_v3.51.0_x5:   }
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: }
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: 0
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5:
2020-12-31T16:04:14.647Z [DEBUG] plugin.terraform-provider-google_v3.51.0_x5: -----------------------------------------------------

从dataflow.googleapis.com返回的错误显然是：

当前用户无法作为服务帐户serviceAccount:[e-mail]进行操作。

此时，我感到困惑的是，尽管Google的数据流API返回了错误，但GCP日志中没有任何指示发生错误的记录。

之后，我有点茅塞顿开。为什么那个错误消息会提到“服务账户serviceAccount”？ 然后我意识到，我定义的服务帐户不正确。 Terraform代码应该是：

resource "google_dataflow_job" "wordcount" {
  # https://stackoverflow.com/a/59931467/201657
  name              = "wordcount"
  template_gcs_path = "gs://dataflow-templates/latest/Word_Count"
  temp_gcs_location = "gs://${local.name-prefix}-functions/temp"
  parameters = {
    inputFile = "gs://dataflow-samples/shakespeare/kinglear.txt"
    output = "gs://${local.name-prefix}-functions/wordcount/output"
  }
  service_account_email = data.google_service_account.sa.email
}

我进行了修正，结果立即生效。是用户自己的错误！

然后，我开始移除添加的各种权限：

gcloud projects remove-iam-policy-binding $PROJECT \
  --member="serviceAccount:dataflowdemo@${PROJECT}.iam.gserviceaccount.com" \
  --role="roles/dataflow.admin"
gcloud projects remove-iam-policy-binding $PROJECT \
  --member="serviceAccount:dataflowdemo@${PROJECT}.iam.gserviceaccount.com" \
  --role="roles/dataflow.worker"

terraform apply 仍然可以工作。然而，在移除角色 roles/dataflow.worker 的授权后，作业失败并出现以下错误：

Workflow failed. Causes: Permissions verification for controller service account failed. IAM role roles/dataflow.worker should be granted to controller service account [e-mail].

因此，关于应该授予的适当角色的文档（https://cloud.google.com/dataflow/docs/concepts/access-control#creating_jobs）是正确的。

如可能显而易见的是，我在知道问题所在之前开始编写这篇文章，我认为将我的调查记录在某个地方可能会有用。现在我已经完成了调查，问题原来是一个 PEBCAK，所以它可能不那么相关了，肯定不应该被接受为答案。尽管如此，在这里留下它可能是有用的信息，关于如何解决 terraform 调用 Google APIs 的问题，并且还重申了所需的角色授权。

- jamiet

Answer 3

我又碰到了这个问题，所以将我的解决方案发布在这里，因为我完全期待在某个时候会再次被这个问题困扰。

我收到了以下错误：

Error: googleapi: Error 403: (a00eba23d59c1fa3): 当前用户无法充当服务帐户[e-mail]。原因：（a00eba23d59c15ac）：当前用户无法充当服务帐户[e-mail]，禁止访问

我正在通过terraform部署dataflow作业，使用另一个服务帐户：[e-mail]

解决方案是授予该服务帐户roles/iam.serviceAccountUser角色：

gcloud projects add-iam-policy-binding myproject \
    --member=serviceAccount:[e-mail] \
    --role=roles/iam.serviceAccountUser

对于那些更喜欢自定义 IAM 角色而不是预定义 IAM 角色的人，缺少的特定权限是iam.serviceAccounts.actAs。

- jamiet

Answer 4

问题已解决！

前往GCP -> 控制台 -> IAM -> 服务帐号电子邮件 -> 添加权限 -> 服务帐号用户，如下所示：

- Deepak Kallungal