e. Auto execute pipeline
In this section, we will update the sample Dockerfile created earlier to automatically trigger the container build and update to Amazon ECR as part of the CodePipeline we created earlier.
We will modify the Dockerfile to run a Genomics workflow using Nextflow.
We will go over the Nextflow architecture and job execution/orchestration more in the next lab. For now, we will go ahead and update the repository and see how the CICD pipeline works for your build.
- First confirm you are in the MyDemoRepo repository:
pwd # should be MyDemoRepo
- Update the Dockerfile to the following. This is an entrypoint script which can consume the link to an Amazon S3 bucket or a git repository from which to download the Nextflow pipeline and executes it.
The Nextflow command-line tool uses the JVM. Thus, we will install AWS open-source variant Amazon Corretto. Amazon Corretto is a no-cost, multiplatform, production-ready distribution of the Open Java Development Kit (OpenJDK). Corretto comes with long-term support that will include performance enhancements and security fixes. Amazon runs Corretto internally on thousands of production services and Corretto is certified as compatible with the Java SE standard. With Corretto, you can develop and run Java applications on popular operating systems, including Linux, Windows, and macOS.
cat > Dockerfile << EOF
FROM public.ecr.aws/amazoncorretto/amazoncorretto:8
RUN curl -s https://get.nextflow.io | bash \
&& mv nextflow /usr/local/bin/
RUN yum install -y git python-pip curl jq
RUN pip install --upgrade awscli
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
VOLUME ["/scratch"]
CMD ["/usr/local/bin/entrypoint.sh"]
EOF
- Copy the entrypoint file (entrypoint.sh) from the S3 bucket and make it an executable
aws s3 cp s3://sc21-hpc-labs/entrypoint.sh .
chmod +x entrypoint.sh
- Now we will update and push this file to the created codecommit repository
git add Dockerfile entrypoint.sh
git commit -m "Updated the Dockerfile to trigger Genomics workflow using Nextflow"
git push origin main
-
In the AWS Management Console search bar, type and select CodePipeline. Click on the MyDemoPipeline that you created in the previous section. You should now see that the CodeCommit push above should have triggered the build via CodeBuild automatically.
-
Click on the Details deep link from the Build stage of the CodePipeline. This will take you to build logs from the CodeBuild project that you created:
-
Click on the Tail logs to see the on-going or completed build process. This is showcasing every step of the build process as provided in your buildspec.yml file.
-
In addition to the build the pipeline is also pushing the built container image to the container registry in Amazon ECR.