在使用Cloud Formation创建Amazon Elasticsearch服务时，出现了CloudWatch资源访问策略错误。

Question

我正在尝试创建一个启用了LogPublishingOptions的弹性搜索域。在启用LogPublishingOptions时，ES说它没有足够的权限在Cloudwatch上创建LogStream。

我尝试使用角色创建策略，并将该策略附加到ES引用的LogGroup，但它不起作用。以下是我的弹性搜索云形成模板：

AWSTemplateFormatVersion: 2010-09-09

Resources:
  MYLOGGROUP:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: index_slow

  MYESROLE:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: es.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AmazonESFullAccess'
        - 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
      RoleName: !Join
        - '-'
        - - es
          - !Ref 'AWS::Region'

  PolicyDocESIndexSlow :
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: 
             - logs:PutLogEvents
             - logs:CreateLogStream
            Resource: 'arn:aws:logs:*'
      PolicyName: !Ref MYLOGGROUP
      Roles:
        - !Ref MYESROLE

  MYESDOMAIN:
    Type: AWS::Elasticsearch::Domain
    Properties:
      DomainName: 'es-domain'
      ElasticsearchVersion: '7.4'
      ElasticsearchClusterConfig:
        DedicatedMasterCount: 3
        DedicatedMasterEnabled: True
        DedicatedMasterType: 'r5.large.elasticsearch'
        InstanceCount: '2'
        InstanceType: 'r5.large.elasticsearch'
      EBSOptions:
        EBSEnabled: True
        VolumeSize: 10
        VolumeType: 'gp2'
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Effect: Deny
            Principal:
              AWS: '*'
            Action: 'es:*'
            Resource: '*'
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: True
      LogPublishingOptions:
        INDEX_SLOW_LOGS:
          CloudWatchLogsLogGroupArn: !GetAtt
            - MYLOGGROUP
            - Arn
          Enabled: True
      VPCOptions:
        SubnetIds:
          - !Ref MYSUBNET
        SecurityGroupIds:
          - !Ref MYSECURITYGROUP
  MYVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
  MYSUBNET:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref MYVPC
      CidrBlock: 10.0.0.0/16
  MYSECURITYGROUP:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: security group for elastic search domain
      VpcId: !Ref MYVPC
      GroupName: 'SG for ES'
      SecurityGroupIngress:
        - FromPort: '443'
          IpProtocol: tcp
          ToPort: '443'
          CidrIp: 0.0.0.0/0

执行时，它会创建除MYESDOMAIN之外的所有资源。它说：

为CloudWatch Logs日志组index_slow指定的资源访问策略未授予足够权限让Amazon Elasticsearch Service创建日志流。请检查资源访问策略。（服务：AWSElasticsearch；状态码：400；错误代码：ValidationException）

这里缺少什么？有任何想法吗？

- Milind Dalvi

Answer 1

2021更新

Amazon Web Services推出了CloudFormation资源 AWS::Logs::ResourcePolicy，用于在CF中定义CloudWatch Logs的策略。我发现的主要问题是它只接受一个真正的字符串作为值。尝试使用Ref，Join等组装字符串会被拒绝。如果有人能够使其工作，那将是很好的。

使用YAML编写比JSON更容易，因为JSON需要转义所有的"字符。

OSLogGroupPolicy:
    Type: AWS::Logs::ResourcePolicy
    Properties:
      PolicyName: AllowES
      PolicyDocument: '{"Version": "2012-10-17","Statement":[{"Effect":"Allow","Principal": {"Service": ["es.amazonaws.com"]},"Action":["logs:PutLogEvents","logs:CreateLogStream"],"Resource":"*"}]}'

- Tobin

---

对于2022年，这是正确的答案。一旦我添加了它，更新就可以完美地进行。如果没有它...我会收到与OP相同的错误。 - sjwoodr

Answer 2

我认为这里存在一些混淆，关于更新/设置哪些策略以启用ES将内容写入日志组。

我认为你应该将PolicyDocESIndexSlow策略应用于CloudWatch Logs。

并且据我所记，在 CloudFormation 中无法完成此操作。您必须使用put-resource-policy相应的API调用或控制台，如下所示:

• 查看 Amazon Elasticsearch Service 慢日志

- Marcin

Answer 3

最终的代码应该是这样的，

部署ES lambda_function.py

import logging
import time

import boto3
import json
from crhelper import CfnResource

logger = logging.getLogger(__name__)
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL', sleep_on_delete=120)

try:
    # Init code goes here
    pass
except Exception as e:
    helper.init_failure(e)


@helper.create
@helper.update
def create(event, _):
    logger.info("Got Create/Update")

    my_log_group_arn = event['ResourceProperties']['MYLOGGROUPArn']

    client = boto3.client('logs')

    policy_document = dict()
    policy_document['Version'] = '2012-10-17'
    policy_document['Statement'] = [{
        'Sid': 'ESLogsToCloudWatchLogs',
        'Effect': 'Allow',
        'Principal': {
            'Service': [
                'es.amazonaws.com'
            ]
        },
        'Action': 'logs:*',
    }]

    policy_document['Statement'][0]['Resource'] = my_log_group_arn 
    client.put_resource_policy(policyName='ESIndexSlowPolicy', policyDocument=json.dumps(policy_document))

    helper.Data['success'] = True
    helper.Data['message'] = 'ES policy deployment successful'

    # To return an error to Cloud Formation you raise an exception:
    if not helper.Data["success"]:
        raise Exception('Error message to cloud formation')

    return "MYESIDDEFAULT"


@helper.delete
def delete(event, _):
    logger.info("Got Delete")
    # Delete never returns anything. Should not fail if the underlying resources are already deleted.
    # Desired state.

    try:
        client = boto3.client('logs')
        client.delete_resource_policy(policyName='ESIndexSlowPolicy')

    except Exception as ex:
        logger.critical(f'ES policy delete failed with error [{repr(ex)}]')


def lambda_handler(event, context):
    helper(event, context)

云形成模板中的一些附加组件

 MYLAMBDAROLE:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
        - 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
        - 'arn:aws:iam::aws:policy/AmazonESFullAccess'
        - 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
      RoleName: !Join
        - '-'
        - - lambda-role
          - !Ref 'AWS::Region'

  MYLAMBDADEPLOY:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        S3Bucket: es-bucket-for-lambda-ta86asdf596
        S3Key: es.zip
      FunctionName: deploy_es
      Handler: lambda_function.lambda_handler
      MemorySize: 128
      Role: !GetAtt
        - MYLAMBDAROLE
        - Arn
      Runtime: python3.8
      Timeout: 60

  MYESSETUP:
    Type: 'Custom::MYESSETUP'
    Properties:
      ServiceToken: !GetAtt
        - MYLAMBDADEPLOY
        - Arn
      MYLOGGROUPArn: !GetAtt
        - MYLOGGROUP
        - Arn
    DependsOn:
      - MYLAMBDADEPLOY
      - MYLOGGROUP

只需在MYESDOMAIN下方添加以下DependsOn

DependsOn:
  - MYESSETUP

- Milind Dalvi

Answer 4

在 AWS::Elasticsearch::Domain 资源中添加 DependsOn，可以解决 AWS::Logs::ResourcePolicy 的问题。

示例代码：

  ESLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Sub '/aws/OpenSearchService/domains/${NamePrefix}-es/application-logs'
      RetentionInDays: 30

  ESLogGroupPolicy:
    Type: AWS::Logs::ResourcePolicy
    DependsOn: ESLogGroup
    Properties:
      PolicyName: !Sub "es-logs-access-policy"
      PolicyDocument: '{"Version": "2012-10-17","Statement":[{"Effect":"Allow","Principal": {"Service": ["es.amazonaws.com"]},"Action":["logs:PutLogEvents","logs:CreateLogStream"],"Resource":"*"}]}'

  ESDomain:
    Type: AWS::Elasticsearch::Domain
    DependsOn: [ESLogGroupPolicy]
    Properties:
      DomainName: !Sub "${NamePrefix}-es"
      ...
      LogPublishingOptions:
        ES_APPLICATION_LOGS:
          CloudWatchLogsLogGroupArn: !GetAtt ESLogGroup.Arn
          Enabled: true

- tedsmitt

Answer 5

对于使用 CDK V3 构造的人：

    const domain = new os.Domain(this, "domain", {...
    domain.addAccessPolicies(
      new iam.PolicyStatement({
        actions: ['logs:CreateLogStream', 'logs:PutLogEvents'],
        resources: [
          `${osAppLogGroup.logGroupArn}/*`,
          `${osSlowSearchLogGroup.logGroupArn}/*`,
          `${osSlowIndexLogGroup.logGroupArn}/*`
        ],
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('es.amazonaws.com')],
      })
    );

- danw