Feature: custom runtimes to define function

[MarlonJD]

Problem

#1543, #1486

Changes

I changed runtime parameter to string from int. It was only nodejs version. Now it can be:

'GO_1_X_PROVIDED_AL2023' | 'NODEJS_16_X' | 'NODEJS_18_X' | 'NODEJS_20_X' | 'PYTHON_3_8' | 'PYTHON_3_9' | 'PYTHON_3_10' | 'PYTHON_3_11' | 'PYTHON_3_12'

It's default NODEJS_18_X same as old functions, people only need to specify runtime: "NODEJS_16_X" if they want another version, it's giving lint error, so it's easy to inform people if it's changed. If it's empty, it's still creating nodejs version, I also add required to entry parameter when it's go or python runtimes.

Edit: I modified defineFunction to props and callback function can be parameters at the same time:

defineFunction({
  name: "my-regular-function",
})

// a go function
defineFunction((scope) => {
  return new GoFunction(scope, "GoFunction", {
    entry: "app/cmd/api",
  })
})

can be usable like proposal in RFC, go function will be usable out of box, python function will be require docker, so python function cannot be used on CI/CD, it can be deploy with custom pipeline on local.

ie: disabling auto deploy, npm ci && export CI=1 && npx ampx pipeline-deploy --branch BRANCH_NAME --app-id AMPLIFY_APP_ID will do the trick for python.

Checklist

• If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
• If this PR requires a change to the Project Architecture README, I have included that update in this PR.
• If this PR requires a docs update, I have linked to that docs PR above.
• If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[changeset-bot]

🦋 Changeset detected

Latest commit: 2fcddbc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@aws-amplify/integration-tests	Minor
@aws-amplify/backend-function	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[edwardfoyle]

Hi @MarlonJD thanks for taking the time to open this. We will likely not merge this in its current form. There are a few reasons for this:

1. We cannot depend on CDK alpha packages in our stable release line. The approach outlined in the RFC you commented on moves the alpha CDK dependency to the customer's project so they are explicitly aware they are using alpha functionality.
2. Changing the runtime parameter from a number union to a string union is a breaking API change which we cannot do without a major version bump of our libraries
3. E2E tests that exercise building and calling python and go functions need to be added
4. The dependency on docker to build the functions needs more consideration of the implications (eg for backend deployments on Amplify hosting and getting started locally)

If you are interested, you can work with @josefaidt to define an interface that is appropriate here and then we can have further discussion on the CDK and docker dependencies.

[MarlonJD]

Hi @edwardfoyle.
1-) I understand what you mean, okay, I'll try to change this
2-) What about adding new parameter without modifying runtime?
3-) e2e tests will be added
4-) python docker build is necessary on python build, if it's not possible ı can remove python in pr

[josefaidt]

Hey @MarlonJD 👋 thanks for taking the time to file this! To add to @edwardfoyle 's note we'd like to avoid taking a dependency on Docker and the alpha distributions of L2 constructs for Go and Python. While those two runtimes are top of mind, the escape hatch is desirable to offer the capability of using defineFunction with any runtime, with an opt-out of TypeScript-based features like typed environment variables and automatic secrets resolution. Would you be interested in pivoting this PR to match the proposal? Happy to discuss any findings/thoughts you have along the way in the RFC!

[MarlonJD]

Hello @josefaidt. I'm trying to change my PR, I'm trying to create defineFunctionWithAnyLanguage in @aws-amplify/backend. Now I'm struggling about creating callback function for Construct/scope and use this function, any help in here? I can create directly with Stack parameter like this:

const lambdaStack = Stack.of(backend.sayHelloHandler.resources.lambda.stack);

const goFunctionStack = new NestedStack(lambdaStack, "say-hello-go-lambda");

export const goFunction = defineFunctionWithAnyLanguage(
  new GoFunction(goFunctionStack, "GoFunction", {
    entry: "./amplify/functions/say-hello-go/",
    runtime: Runtime.PROVIDED_AL2023,
    environment: {
      GOARCH: "amd64",
    },
    bundling: {
      goBuildFlags: ["-tags lambda.norpc"],
    },
  });

But it's will depend to backend variable in backend.ts file. I think it's bad idea, defineFunction is using new Error().stack as callerStack in FunctionFactory. Should I use same new Error().stack as a scope of defineFunctionWithAnyLanguage ?

[josefaidt]

Hey @MarlonJD you can define a union on the defineFunction argument to either take the existing props type or a callback like (scope: Construct) => lambda.Function, but this scope will be passed at buildtime from the defineFunction factory.

the props here will need adjustments
https://github.com/aws-amplify/amplify-backend/blob/main/packages/backend-function/src/factory.ts#L41
https://github.com/aws-amplify/amplify-backend/blob/main/packages/backend-function/src/factory.ts#L101

and the scope would be passed to the AmplifyFunction construct
https://github.com/aws-amplify/amplify-backend/blob/main/packages/backend-function/src/factory.ts#L235

We wouldn't want to necessarily change the scope but change how "props" (in this case, the callback) is called with the existing scope to initialize within the Amplify context

[MarlonJD]

@josefaidt I changed PR to this:

 const lambda = defineFunction({
  entry: './test-assets/default-lambda/handler.ts',
  runtime: 16,
});

or

const lambda = defineFunction((scope) => {
  return new NodejsFunction(scope, 'customFunction', {
    entry:
      './packages/backend-function/src/test-assets/default-lambda/handler.ts',
  });
});

Golang function will be usable out of box, python function will be require docker, so python function cannot be used on CI/CD, it can be deploy with custom pipeline on local.

ie: disabling auto deploy, npm ci && export CI=1 && npx ampx pipeline-deploy --branch BRANCH_NAME --app-id AMPLIFY_APP_ID will do the trick for python.

What are you thinking about this changes? If it's okay, I'll try to write e2e tests, can you guide me again?

[MarlonJD]

I hope you can review this changes @edwardfoyle

[edwardfoyle]

This is heading in the right direction! Another thing we need to do is turn off generating the typed process.env file when using custom runtimes. We can probably do that by creating a new noop FunctionEnvironmentTypeGenerator if a callback is specified to defineFunction

We should also have an integration and/or e2e test that asserts permissions and environment variables are correctly attached to functions with custom runtimes.

[MarlonJD]

@edwardfoyle I did changes, if it's okay about code, can you help me how and where should write for integration and/or e2e tests?

[MarlonJD]

@edwardfoyle Hey there, I hope you can review new changes

[edwardfoyle]

Hi @MarlonJD, sorry for the delay here. While this PR is on the right track in terms of implementation, we are concerned about the implications when deploying using the Amplify hosting service. Specifically, CDK requires Docker to build some runtimes and currently Docker is not installed in the hosting build container. Docker also cannot be installed using a custom build script because it requires permissions that the execution environment doesn't have. This means that customers using custom runtimes would not be able to use Hosting deployments. We want to consider this feature holistically rather than only offer partial support across our product offering. As a result, we need to do some internal prerequisite work to ensure custom runtime deployments can work in Amplify hosting deployments.

[MarlonJD]

@josefaidt @edwardfoyle Hey there! I have good news about python functions. There is already a local paramater in BundlingOptions. It gives us to bundle without docker. I just found a way to bypass docker build without a modify packages, here is the usage:

import { defineFunction } from "@aws-amplify/backend";
import { DockerImage, Duration } from "aws-cdk-lib";
import { Code, Function, Runtime } from "aws-cdk-lib/aws-lambda";
import { execSync } from "child_process";
import * as path from "path";

const functionDir = path.join(__dirname);

export const sayHelloPythonFunctionHandler = defineFunction(
  (scope) =>
    new Function(scope, "say-hello-python-xderawfe", {
      handler: "index.handler",
      runtime: Runtime.PYTHON_3_9,
      timeout: Duration.seconds(20),
      code: Code.fromAsset(functionDir, {
        bundling: {
          image: DockerImage.fromRegistry("dummy"),
          local: {
            tryBundle(outputDir: string) {
              // if you want to use another python version, make default this ie: pyvenv global 3.12
              // if you dont have python3.9 installed, you can use pyenv to install it from Amplify Hosting build settings
              // from amplify.yaml, you can add this command to install python3.9 in the build settings
              // ie: pyenv install 3.12, pyenv global 3.12

              // Add pyenv command here, if you have more than one python version installed in your functions

              execSync(
                `python3 -m pip install -r ${path.join(functionDir, "requirements.txt")} -t ${path.join(outputDir)} --platform manylinux2014_x86_64 --only-binary=:all:`
              );
              execSync(`rsync -rLv ${functionDir}/* ${path.join(outputDir)}`);
              return true;
            },
          },
        },
      }),
    })
);

I just tried any it's works. I used pure Function, for this reason we can use any language supported from Lambda, f you can bundle it correctly without any package use.
We can add samples to docs how people can use functions.
Now Go and Python functions ready to use, I hope we can move on to the PR and tests.

[MarlonJD]

@edwardfoyle Any updates?

[MarlonJD]

PTAL

[Srakai]

Great work @MarlonJD! Hope this gets merged quickly as currently im trying to achieve same but in a really whacky way

[MarlonJD]

Hey @josefaidt, I commented how we can use python functions without docker build, but there is no local parameter in PythonFunction in @aws-cdk/aws-lambda-python-alpha, so I just showed example on Pure Function in aws-cdk-lib/aws-lambda to create python function with local parameter, it's enough to use python function.

What are you thinking about this solution to using python functions like this, should we add local parameter to @aws-cdk/aws-lambda-python-alpha or is it okay to use like this?

[mrubelmann]

Any updates on this PR? @edwardfoyle, has @MarlonJD addressed your request? It would be super, super helpful to get this merged!

[guikcd]

@josefaidt @edwardfoyle Hey there! I have good news about python functions. There is already a local paramater in BundlingOptions. It gives us to bundle without docker. I just found a way to bypass docker build without a modify packages, here is the usage:

import { defineFunction } from "@aws-amplify/backend";
import { DockerImage, Duration } from "aws-cdk-lib";
import { Code, Function, Runtime } from "aws-cdk-lib/aws-lambda";
import { execSync } from "child_process";
import * as path from "path";

const functionDir = path.join(__dirname);

export const sayHelloPythonFunctionHandler = defineFunction(
  (scope) =>
    new Function(scope, "say-hello-python-xderawfe", {
      handler: "index.handler",
      runtime: Runtime.PYTHON_3_9,
      timeout: Duration.seconds(20),
      code: Code.fromAsset(functionDir, {
        bundling: {
          image: DockerImage.fromRegistry("dummy"),
          local: {
            tryBundle(outputDir: string) {
              // if you want to use another python version, make default this ie: pyvenv global 3.12
              // if you dont have python3.9 installed, you can use pyenv to install it from Amplify Hosting build settings
              // from amplify.yaml, you can add this command to install python3.9 in the build settings
              // ie: pyenv install 3.12, pyenv global 3.12

              // Add pyenv command here, if you have more than one python version installed in your functions

              execSync(
                `python3 -m pip install -r ${path.join(functionDir, "requirements.txt")} -t ${path.join(outputDir)} --platform manylinux2014_x86_64 --only-binary=:all:`
              );
              execSync(`rsync -rLv ${functionDir}/* ${path.join(outputDir)}`);
              return true;
            },
          },
        },
      }),
    })
);

I just tried any it's works. I used pure Function, for this reason we can use any language supported from Lambda, f you can bundle it correctly without any package use. We can add samples to docs how people can use functions. Now Go and Python functions ready to use, I hope we can move on to the PR and tests.

I was able to build my python custom function with my requirements. The only problem I have now is that this seems to be never updated even if I alter this line. Seems that a missing force/cache somewhere. Only workaround I've found now: remove the .amplify directory to force the re-generation of the bundle.

[MarlonJD]

@guikcd oh it's a good review, let me check how nodejs changes triggers, maybe we can found fix about it

[MarlonJD]

@josefaidt Hey there, I was looking into tracking changes on lambda functions, but I couldn't found this yet, I found FilesChangesTracker class but it's for sandbox file changes, can you help me how can I found how lambda functions triggering bundling again when function have changes?

[josefaidt]

Hey folks in the thread 👋 we're working towards a few things to smooth over the DX for this feature's prerequisites -- namely Docker builds in Amplify Hosting and custom synthesizer support.

@MarlonJD to your latest comment, the file watcher you will need to expand the watcher class to allow reacting to changes from files with extensions that are not ts, tsx

[MarlonJD]

Hey @josefaidt, I wonder that why we we can't merge right now? Go functions are just ready for deploy, and also we can use python without docker (#1602 (comment)), go and python libraries are alpha so we can make some changes for default building for Linux in CI (#1602 (comment)) What's your ideas about these

[arash2060]

This is great and I was actually able to deploy a python lambda function using this. However, I don't see how this approach can be integrated with amplify secrets. Am I missing something?

Context: within FunctionProps, we are able to do something like this:

environment: {
        SOURCE_ADDRESS: 'sourceAddress',
        OPEN_AI_KEY: secret('OPEN_AI_KEY'),
    },
    

[MarlonJD]

This is great and I was actually able to deploy a python lambda function using this. However, I don't see how this approach can be integrated with amplify secrets. Am I missing something?

Context: within FunctionProps, we are able to do something like this:

environment: {
        SOURCE_ADDRESS: 'sourceAddress',
        OPEN_AI_KEY: secret('OPEN_AI_KEY'),
    },
    

I'm not sure but as I understand from the code, secrets only usable from CI from amplify itself without manuel deploy,
Did you push with manual deploy?
I used .env files to use secret values as a workaround

[arash2060]

Following up on my overall comment, I'm not sure how we're going to handle environment variables of type Record<string, string | BackendSecret>;. (see line 88 of this file) I see that you can add an environment variable via addEnvironment(key: string, value: string, options?: EnvironmentOptions): this; method of PythonFunction. However, the value cannot be of type BackendSecret