AWS CodePipeline：将Lambda函数的输出传递给CloudFormation

Question

我想使用CodePipeline运行一个CloudFormation模板。这个模板需要一个输入参数，其中包含当前日期/时间。不幸的是，CloudFormation无法自己生成当前日期时间。

我的方法首先是运行一个简单的Lambda函数来创建当前时间戳并将其保存为OutputArtifacts。然后CloudFormation任务导入此工件作为InputArtifacts，并从DateTime属性获取值，通过ParameterOverrides指令将其传递给CloudFormation。

不幸的是，CodePipeline一直报“DateTimeInput”参数无效（显然GetArtifactAtt查找失败）。我认为lambda输出（python：print）没有正确保存为工件？

您知道如何正确传递lambda输出，或者您有更好的实现方法吗？

所有管道组件均使用YAML定义为CloudFormation。以下是相关部分：

Lambda函数：

Resources:
  ...
  GetDateTimeFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Runtime: python2.7
      Timeout: '10'
      Role: !GetAtt GetDateTimeFunctionExecutionRole.Arn
      Code:
        ZipFile: |
                import datetime
                import boto3
                import json

                code_pipeline = boto3.client('codepipeline')

                def lambda_handler(event, context):
                  now = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
                  responseData = {'DateTime':now}
                  print json.dumps(responseData)
                  response = code_pipeline.put_job_success_result(jobId=event['CodePipeline.job']['id'])
                  return response

以下是管道任务：

Resources:
...
  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtifactStore:
        Location: !Ref ArtifactStoreBucket
        Type: S3
      DisableInboundStageTransitions: []
      Name: !Ref PipelineName
      RoleArn: !GetAtt PipelineRole.Arn
      Stages:
        - Name: Deploy
          Actions:
            - Name: GetDateTime
              RunOrder: 1
              ActionTypeId:
                Category: Invoke
                Owner: AWS
                Provider: Lambda
                Version: '1'
              Configuration:
                 FunctionName: !Ref GetDateTimeFunction
              OutputArtifacts:
                - Name: GetDateTimeOutput
            - Name: CreateStack
              RunOrder: 2
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: '1'
              InputArtifacts:
                - Name: TemplateSource
                - Name: GetDateTimeOutput
              Configuration:
                ActionMode: REPLACE_ON_FAILURE
                Capabilities: CAPABILITY_IAM
                RoleArn: !GetAtt CloudFormationRole.Arn
                StackName: !Ref CFNStackname
                TemplatePath: !Sub TemplateSource::${CFNScriptfile}
                TemplateConfiguration: !Sub TemplateSource::${CFNConfigfile}
                ParameterOverrides: |
                  {
                    "DateTimeInput" : { "Fn::GetArtifactAtt" : [ "GetDateTimeOutput", "DateTime" ] }
                  }

更新：我太天真了，认为会有一个简单的方法。现在我知道，仅仅提供一个简单的Lambda输出工件就是一项更高级和手动的任务。

在Python代码内部，必须评估传递的event字典（CodePipeline.job），以查找：
- 预定义的OutputArtifacts（S3 Bucket/Key）和
- CodePipeline提供的临时S3会话凭据。
然后，需要用这些凭证初始化S3客户端，并运行S3 put_object。

https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html https://forums.aws.amazon.com/thread.jspa?threadID=232174

所以我的问题又来了：你们有什么更好或更简单的方法来实现这一点吗？
我只想将当前日期和时间作为CloudFormation的输入参数，而不想破坏自动化。

- mengmann

Answer 1

是的，我没有意识到需要手动处理输出的工件。

最后这个方法解决了问题：

  GetDateTimeFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Runtime: python2.7
      Timeout: '10'
      Role: !GetAtt GetDateTimeFunctionExecutionRole.Arn
      Code:
        ZipFile: |
                from __future__ import print_function
                from boto3.session import Session
                from zipfile import ZipFile
                import json
                import datetime
                import boto3
                import botocore
                import traceback
                import os
                import shutil


                code_pipeline = boto3.client('codepipeline')

                def evaluate(event):
                    # Extract attributes passed in by CodePipeline
                    job_id = event['CodePipeline.job']['id']
                    job_data = event['CodePipeline.job']['data']

                    config = job_data['actionConfiguration']['configuration']
                    credentials = job_data['artifactCredentials']

                    output_artifact = job_data['outputArtifacts'][0]
                    output_bucket = output_artifact['location']['s3Location']['bucketName']
                    output_key = output_artifact['location']['s3Location']['objectKey']

                    # Temporary credentials to access CodePipeline artifact in S3
                    key_id = credentials['accessKeyId']
                    key_secret = credentials['secretAccessKey']
                    session_token = credentials['sessionToken']

                    return (job_id, output_bucket, output_key, key_id, key_secret, session_token)

                def create_artifact(data):
                    artifact_dir = '/tmp/output_artifacts/'+str(uuid.uuid4())
                    artifact_file = artifact_dir+'/files/output.json'
                    zipped_artifact_file = artifact_dir+'/artifact.zip'
                    try:
                        shutil.rmtree(artifact_dir+'/files/')
                    except Exception:
                        pass
                    try:
                        os.remove(zipped_artifact_file)
                    except Exception:
                        pass
                    os.makedirs(artifact_dir+'/files/')
                    with open(artifact_file, 'w') as outfile:
                        json.dump(data, outfile)
                    with ZipFile(zipped_artifact_file, 'w') as zipped_artifact:
                        zipped_artifact.write(artifact_file, os.path.basename(artifact_file))

                    return zipped_artifact_file

                def init_s3client (key_id, key_secret, session_token):
                    session = Session(aws_access_key_id=key_id, aws_secret_access_key=key_secret, aws_session_token=session_token)
                    s3client = session.client('s3', config=botocore.client.Config(signature_version='s3v4'))
                    return s3client

                def lambda_handler(event, context):
                    try:
                        (job_id, output_bucket, output_key, key_id, key_secret, session_token)=evaluate(event)
                        (s3client)=init_s3client(key_id, key_secret, session_token)

                        now=datetime.datetime.now().strftime('%Y-%m-%d_%H:%M:%S')
                        data={"DateTime":now}

                        (zipped_artifact_file)=create_artifact(data)

                        s3client.upload_file(zipped_artifact_file, output_bucket, output_key, ExtraArgs={"ServerSideEncryption": "AES256"})

                        # Tell CodePipeline we succeeded
                        code_pipeline.put_job_success_result(jobId=job_id)

                    except Exception as e:
                        print("ERROR: " + repr(e))
                        message=repr(e)
                        traceback.print_exc()
                        # Tell CodePipeline we failed
                        code_pipeline.put_job_failure_result(jobId=job_id, failureDetails={'message': message, 'type': 'JobFailed'})

                    return "complete"


  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtifactStore:
        Location: !Ref ArtifactStoreBucket
        Type: S3
      DisableInboundStageTransitions: []
      Name: !Ref PipelineName
      RoleArn: !GetAtt PipelineRole.Arn
      Stages:
        - Name: S3Source
          Actions:
            - Name: TemplateSource
              RunOrder: 1
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: S3
                Version: '1'
              Configuration:
                S3Bucket: !Ref ArtifactStoreBucket
                S3ObjectKey: !Ref SourceS3Key
              OutputArtifacts:
                - Name: TemplateSource
        - Name: Deploy
          Actions:
            - Name: GetDateTime
              RunOrder: 1
              ActionTypeId:
                Category: Invoke
                Owner: AWS
                Provider: Lambda
                Version: '1'
              Configuration:
                 FunctionName: !Ref GetDateTimeFunction
              OutputArtifacts:
                - Name: GetDateTimeOutput
            - Name: CreateStack
              RunOrder: 2
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: '1'
              InputArtifacts:
                - Name: TemplateSource
                - Name: GetDateTimeOutput
              Configuration:
                ActionMode: REPLACE_ON_FAILURE
                Capabilities: CAPABILITY_IAM
                RoleArn: !GetAtt CloudFormationRole.Arn
                StackName: !Ref CFNStackname
                TemplatePath: !Sub TemplateSource::${CFNScriptfile}
                TemplateConfiguration: !Sub TemplateSource::${CFNConfigfile}
                ParameterOverrides: |
                  {
                    "DateTimeInput" : { "Fn::GetParam" : ["GetDateTimeOutput", "output.json", "DateTime"]}
                  }

这么微不足道的任务，开销有点大啊 ;-）

- mengmann

Answer 2

您应该使用"Fn::GetParam"而不是"Fn::GetArtifactAtt"。根据CloudFormation文档，"Fn::GetArtifactAtt"只能获取artifact的属性，如BucketName、ObjectKey和URL。"Fn::GetParam"可以从artifact中的json文件中获取值。因此，如果您可以将artifact“GetDateTimeOutput”生成为zip文件，其中包括一个JSON文件（例如param.json），其内容如下：

{ "DateTime": "2018/10/31 13:32:00" }

那么您可以使用{ "Fn::GetParam" : [ "GetDateTimeOutput", "param.json", "DateTime" ] }来获取时间。

您可以修改Lambda函数来执行此操作，也可以使用CodeBuild操作。CodeBuild负责创建zip文件，您只需要指定构建命令以在输出文件夹中创建JSON文件即可。您可以在以下文档中找到有关如何在CodePipeline中使用CodeBuild的更多信息。

CloudFormation文档 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-parameter-override-functions.html#w2ab1c13c17b9

CodeBuild文档 https://docs.aws.amazon.com/codebuild/latest/userguide/how-to-create-pipeline.html

- zhngha