Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIPS #267

Merged
merged 34 commits into from
Mar 12, 2024
Merged

FIPS #267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
b4e75ab
omg, we need a flag
TingDaoK Jan 5, 2024
1ba95e0
wops
TingDaoK Jan 5, 2024
1a4b484
golang for ubuntu?
TingDaoK Jan 5, 2024
52fd4eb
golang1.18
TingDaoK Jan 8, 2024
62a1729
install golang
TingDaoK Jan 8, 2024
2aa26dc
install go with builder
TingDaoK Jan 8, 2024
d015a16
use distutils
TingDaoK Jan 8, 2024
10e3e39
don't even use distutils
TingDaoK Jan 8, 2024
6bf5b52
openbsd
TingDaoK Jan 8, 2024
73b3d3c
set the flag after install
TingDaoK Jan 8, 2024
cc3da23
this?
TingDaoK Jan 8, 2024
b920ac1
windows and mac
TingDaoK Jan 8, 2024
70899c2
install perl
TingDaoK Jan 9, 2024
ddf1107
remove unused part
TingDaoK Jan 11, 2024
86287bc
as binary
TingDaoK Jan 11, 2024
1be485c
I don't know
TingDaoK Jan 11, 2024
151a39c
what about this?
TingDaoK Jan 11, 2024
10efcb7
I don't know////
TingDaoK Jan 11, 2024
62e673e
something weird and wrong with ubuntu docker image, I am done trying …
TingDaoK Jan 11, 2024
869b82e
switch to fips of awslc entirely
TingDaoK Jan 19, 2024
0ea4c63
install perl
TingDaoK Feb 16, 2024
8cf8ebc
Merge branch 'main' into fips
TingDaoK Feb 16, 2024
dc2588b
perl-strictures ?
TingDaoK Feb 16, 2024
130e29d
perl-strictures it is
TingDaoK Feb 16, 2024
ff91fc0
don't know
TingDaoK Feb 17, 2024
962457f
to support cross compile
TingDaoK Feb 19, 2024
422dcb3
use the old way to set the path
TingDaoK Feb 19, 2024
10356a2
remove unused stuff
TingDaoK Feb 19, 2024
5805dd2
format
TingDaoK Feb 19, 2024
3c25475
set the go path from cmake
TingDaoK Feb 19, 2024
c1667df
format
TingDaoK Feb 19, 2024
1eb95bd
check if it exist first
TingDaoK Feb 20, 2024
d52a45f
waht?
TingDaoK Feb 20, 2024
e9bb3ee
Merge branch 'main' into fips
TingDaoK Mar 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/docker-images/alpine-3.16-arm64/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ RUN apk add \
ca-certificates \
cmake \
bash \
aws-cli
aws-cli \
perl-strictures
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this weird specific package?

https://metacpan.org/pod/strictures

strictures - Turn on strict and make most warnings fatal

Is there something more basic, named like perl?

Copy link
Contributor Author

@TingDaoK TingDaoK Mar 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, but aws-lc needs strict module for perl.

anyway, we are not turning fips on by default now. I can remove this


WORKDIR /tmp

Expand Down
3 changes: 2 additions & 1 deletion .github/docker-images/alpine-3.16-x64/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ RUN apk add \
ca-certificates \
cmake \
bash \
aws-cli
aws-cli \
perl-strictures

WORKDIR /tmp

Expand Down
4 changes: 2 additions & 2 deletions .github/docker-images/openwrt-x64-openjdk8/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ RUN mkdir -p /usr/local/bin
RUN opkg update

# packages in openwrt
RUN opkg install git-http ca-bundle curl python3 python3-pip gcc make bash sudo
RUN opkg install git-http ca-bundle curl python3 python3-pip gcc make bash sudo perl

# packages we have to get from alpine

Expand All @@ -27,7 +27,7 @@ RUN tar -xzf apk-tools-static-2.12.9-r3.apk
RUN ./sbin/apk.static -X http://dl-cdn.alpinelinux.org/alpine/v3.16/main -X http://dl-cdn.alpinelinux.org/alpine/v3.16/community -U --allow-untrusted --initdb add cmake openjdk8 maven aws-cli

# stub libraries for stuff we unconditionally link; functionality is all actually in musl already
# long term we might want to make our recognition better, but this is a blocker for the s2n build
# long term we might want to make our recognition better, but this is a blocker for the s2n build
RUN ar -rc /usr/lib/libpthread.a
RUN ar -rc /usr/lib/libdl.a
RUN ar -rc /usr/lib/librt.a
Expand Down
6 changes: 6 additions & 0 deletions .github/docker-images/ubuntu-18-x64/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ RUN apt-get update -qq \
ca-certificates \
&& apt-get clean

# Add the longsleep/golang-backports PPA
RUN apt-get update && apt-get install -y software-properties-common && add-apt-repository ppa:longsleep/golang-backports && apt-get update
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debatable: I would rather investigate the failure on x64 instead of having two different methods for installing GO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did spend half a day to investigate it, and I don't know.

The issue is when we extract the go.tar.gz something failed. I tried different packages, and different ways to extract, no luck.

It only happens for ubuntu, and I gave up


# Install Go from the PPA
RUN apt-get install -y golang-go

###############################################################################
# Python/AWS CLI
###############################################################################
Expand Down
6 changes: 6 additions & 0 deletions .github/docker-images/ubuntu-20-x64/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ RUN apt-get update -qq \
ca-certificates \
&& apt-get clean

# Add the longsleep/golang-backports PPA
RUN apt-get update && apt-get install -y software-properties-common && add-apt-repository ppa:longsleep/golang-backports && apt-get update

# Install Go from the PPA
RUN apt-get install -y golang-go

###############################################################################
# Python/AWS CLI
###############################################################################
Expand Down
1 change: 1 addition & 0 deletions .github/docker-images/ubuntu-22-x64/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ RUN apt-get update -qq \
software-properties-common \
apt-transport-https \
ca-certificates \
golang-go \
&& apt-get clean

###############################################################################
Expand Down
11 changes: 7 additions & 4 deletions builder/actions/cmake.py
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,12 @@ def _project_dirs(env, project):
def _build_project(env, project, cmake_extra, build_tests=False, args_transformer=None, coverage=False):
sh = env.shell
config = project.get_config(env.spec)
build_env = []
toolchain = env.toolchain
if toolchain.cross_compile and 'go_path' in env.variables:
# We need to set the envrionment variable of GO_PATH for cross compile
build_env = ["GO_PATH={}\n".format(env.variables['go_path'])]

# build dependencies first, let cmake decide what needs doing
for dep in project.get_dependencies(env.spec):
_build_project(env, dep, cmake_extra)
Expand Down Expand Up @@ -178,13 +183,11 @@ def _build_project(env, project, cmake_extra, build_tests=False, args_transforme
cmake_args = args_transformer(env, project, cmake_args)

# When cross compiling, we must inject the build_env into the cross compile container
build_env = []
if toolchain.cross_compile:
build_env = ['{}={}\n'.format(key, val)
for key, val in config.get('build_env', {}).items()]
build_env = build_env + ['{}={}\n'.format(key, val)
for key, val in config.get('build_env', {}).items()]
with open(toolchain.env_file, 'a') as f:
f.writelines(build_env)

# set parallism via env var (cmake's --parallel CLI option doesn't exist until 3.12)
if os.environ.get('CMAKE_BUILD_PARALLEL_LEVEL') is None:
sh.setenv('CMAKE_BUILD_PARALLEL_LEVEL', str(os.cpu_count()))
Expand Down
94 changes: 94 additions & 0 deletions builder/imports/golang.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0.

import os
from pathlib import Path

from builder.core.fetch import fetch_and_extract, mirror_package
from builder.core.project import Import
import builder.core.util as util
from builder.core.host import current_platform

URLs = {
'linux-armv6': 'https://go.dev/dl/go1.21.5.linux-armv6l.tar.gz',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any specific reason for 1.25.5?

if we just want latest versions, looks like we can them up here https://go.dev/dl/?mode=json

'linux-armv7': 'https://go.dev/dl/go1.21.5.linux-armv6l.tar.gz',
'linux-armv8': 'https://go.dev/dl/go1.21.5.linux-arm64.tar.gz',
'linux-x86': 'https://go.dev/dl/go1.21.5.linux-386.tar.gz',
'linux-x64': 'https://go.dev/dl/go1.21.5.linux-amd64.tar.gz',
'openbsd-x64': 'https://go.dev/dl/go1.21.5.linux-amd64.tar.gz',
'windows-x64': 'https://go.dev/dl/go1.21.5.windows-amd64.zip',
'windows-x86': 'https://go.dev/dl/go1.21.5.windows-386.zip',
'macos-x64': 'https://go.dev/dl/go1.21.5.darwin-amd64.tar.gz',
}


class GOLANG(Import):
def __init__(self, **kwargs):
super().__init__(
config={},
**kwargs)
self.path = None
self.installed = False

def resolved(self):
return True

def install(self, env):
if self.installed:
return

sh = env.shell

target = '{}-{}'.format(env.spec.target, env.spec.arch)

cross_compile = util.deep_get(env, 'toolchain.cross_compile', False)

# If this is a local build, check the local machine
if not cross_compile or target not in URLs:
# run `go version`
result = util.run_command('go', 'version')
if result.returncode == 0:
# check the version, we need version >=1.18
version_str = result.output.split(" ")[2][2:]
version_numbers = list(map(int, version_str.split('.')))
compare_version_numbers = list(map(int, "1.18.0".split('.')))
if version_numbers >= compare_version_numbers:
return

if target not in URLs:
raise EnvironmentError(
'No pre-built binaries for {} are available, please install golang greater than 1.18'.format(target))

install_dir = os.path.join(env.deps_dir, self.name.lower())
# If path is going to be relative, it has to be relative to the source directory
self.path = str(Path(install_dir).relative_to(env.root_dir))
print('Installing pre-built golang binaries for {} to {}'.format(
target, install_dir))

sh.mkdir(install_dir)
if cross_compile:
# If cross compile using the go execuble for current platform instead to codegen
url = URLs[current_platform()]
else:
url = URLs[target]
ext = '.tar.gz' if url.endswith('.tar.gz') else '.zip'
filename = '{}/golang{}'.format(install_dir, ext)
print('Downloading {}'.format(url))
fetch_and_extract(url, filename, install_dir)
os.remove(filename)

# Set PATH
if cross_compile:
# Path to go binary
env.variables['go_path'] = "/work/"+str(Path(os.path.join(install_dir, 'go/bin')
).relative_to(env.root_dir))
else:
# export the PATH directly if not cross compile.
# env.variables['go_path'] = '{}/go/bin'.format(install_dir)
sh.setenv('PATH', '{}{}{}'.format('{}/go/bin'.format(install_dir), os.pathsep, sh.getenv('PATH')))

self.installed = True

def mirror(self, env):
for src_url in URLs.values():
mirror_package(self.name, src_url)
Loading