Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate fastly log processor to pluto #388

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from
Draft

Migrate fastly log processor to pluto #388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b2a28cd
Typo
edolstra Feb 22, 2024
230f4ca
Turn flake.nix into a module
edolstra Feb 22, 2024
0d98ce1
Give the fastly-log-processor user access to the Fastly logs
edolstra Feb 22, 2024
5256d86
pluto: Enable the Fastly log processor
edolstra Feb 22, 2024
2e71b2c
Use the fastly-log-processor user
edolstra Feb 22, 2024
bd43518
Merge remote-tracking branch 'origin/master' into migrate-fastly-log-…
edolstra Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions delft/pluto/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
../../modules/netboot-serve.nix
../../modules/rfc39.nix
../../modules/tarball-mirror.nix
../../metrics/fastly/process-raw-nix-logs.nix
];

networking = {
Expand Down
8 changes: 4 additions & 4 deletions metrics/fastly/cron.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
#! /bin/sh -e

export AWS_PROFILE=nixos-org
export AWS_PROFILE=fastly-log-processor

now=$(date +%s)
#now=$((now - 86400))
prev_week=$((($now / 86400 / 7)))

from_date_incl=$(date +%F --date="@$(($prev_week * 86400 * 7 - 2 * 86400))")
to_date_incl=$(date +%F --date="@$(($prev_week * 86400 * 7 + 5 * 86400))")
to_date_excl=$(date +%F --date="@$(($prev_week * 86400 * 7 + 5 * 86400))")

echo "Ingesting [$from_date_incl, $to_date_incl)."
echo "Ingesting [$from_date_incl, $to_date_excl)."

marker="$HOME/weeks-done/$prev_week"

Expand All @@ -21,6 +21,6 @@ fi
mkdir -p "$(dirname "$marker")"
touch $marker

./ingest-raw-logs.sh "$from_date_incl" "$to_date_incl"
./ingest-raw-logs.sh "$from_date_incl" "$to_date_excl"

./run-queries.sh
28 changes: 0 additions & 28 deletions metrics/fastly/flake.nix
View file
Open in desktop

This file was deleted.

25 changes: 25 additions & 0 deletions metrics/fastly/process-raw-nix-logs.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{ config, pkgs, ... }: {

users.users.nix-metrics =
{ isNormalUser = true;
description = "Nix Metrics Collection";
};
Comment on lines +3 to +6
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a statically allocated user (and uid), or can the service use the DynamicUser feature, which creates an ad hoc user, when the service runs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's probably possible. But it does need to have the AWS credentials somewhere.

Copy link
Member

@mweinelt mweinelt Feb 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In which case we could probably use LoadCredential.

   systemd.services.process-raw-nix-logs.serviceConfig.LoadCredential = [
     "aws-cred:/path/to/actual/aws-cred-file"
   ];

The secret will then be provided at $CREDENTIALS_DIRECTORY/aws-cred.

We currently deploy secrets through agenix. I can help integrate the secret, if needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can share the secret with me via Bitwarden and I'll create the age file? Feel free to reach out on Matrix to expedite this process.


systemd.services.process-raw-nix-logs = {
description = "Process Raw nixos.org Logs";
serviceConfig.Type = "oneshot";
serviceConfig.User = "nix-metrics";
path =
[ pkgs.awscli
pkgs.jq
];
script =
''
cd ${./.}
./cron.sh
'';
startAt = "Tue 07:30";
};

}

22 changes: 10 additions & 12 deletions terraform-iam/fastlylog/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,19 +31,17 @@ resource "aws_s3_bucket_policy" "logs" {
"Sid": "AllowNixOSOrgRead",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::008826681144:user/eelco.dolstra"
"AWS": "arn:aws:iam::008826681144:user/eelco.dolstra",
"AWS": "arn:aws:iam::008826681144:user/fastly-log-processor"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::${aws_s3_bucket.logs.id}/*"
},
{
"Sid": "AllowNixOSOrgList",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::008826681144:user/eelco.dolstra"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::${aws_s3_bucket.logs.id}"
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::${aws_s3_bucket.logs.id}/*",
"arn:aws:s3:::${aws_s3_bucket.logs.id}"
]
}
]
}
Expand Down
Loading