Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aws #227

Open
wants to merge 154 commits into
base: aws
Choose a base branch
from
Open

Aws #227

Changes from 1 commit
Commits
Show all changes
154 commits
Select commit Hold shift + click to select a range
d279ab0
Merge pull request #20 from chpc-tech-eval/dev
nyameko Jun 20, 2024
3075a2e
Merge pull request #21 from chpc-tech-eval/stag
nyameko Jun 20, 2024
f48b191
Merge pull request #25 from chpc-tech-eval/dev
nyameko Jun 29, 2024
893e565
Modified listing of installs.
nyameko Jul 6, 2024
65860d1
Updated tut 1 with imagery. and tut 2 descriptions.
nyameko Jul 6, 2024
ddef9e9
Merge pull request #101 from chpc-tech-eval/tut2/visual_clarity
nyameko Jul 6, 2024
728227e
Updated networking sections.
nyameko Jul 6, 2024
d7eebba
Fixed Instance Flavors -> naming.
nyameko Jul 6, 2024
2eff555
Added troubleshooting images.
nyameko Jul 6, 2024
973ea72
Added Stateful Firewall.
nyameko Jul 7, 2024
a5ab4ec
Merge branch 'dev' into tut2/visual_clarity
nyameko Jul 7, 2024
55cc6c9
Merge pull request #103 from chpc-tech-eval/tut2/visual_clarity
nyameko Jul 7, 2024
2b13428
Incremental update
nyameko Jul 7, 2024
5a6aca9
Tidied up NTP.
nyameko Jul 7, 2024
5e6f6c9
Completed working on ntp and nfs.
nyameko Jul 7, 2024
86da0d6
Merge pull request #104 from chpc-tech-eval/tut2/ntp_nfs
nyameko Jul 7, 2024
08cf88c
Updated TOC
nyameko Jul 7, 2024
9897822
Removed comment which was captured as heading
nyameko Jul 7, 2024
af5a91a
Updated toc and completed passwordless.
nyameko Jul 7, 2024
1ca650a
Left with Ansible and Wireguard.
nyameko Jul 7, 2024
c58c3ae
Merge pull request #105 from chpc-tech-eval/tut2/passwordless_ansible…
nyameko Jul 7, 2024
32707d3
Added JupyterLab sections to Tutorial 4.
nyameko Jul 7, 2024
a72e1ae
Updated TOC.
nyameko Jul 7, 2024
83fd9f6
Fixed port from 8080 -> 8889
nyameko Jul 7, 2024
652953a
Merge pull request #115 from chpc-tech-eval/tut3/closing_issues
nyameko Jul 7, 2024
69cc0d0
CHPC 2024 Student Cluster Competition Selection Round.
nyameko Jul 8, 2024
b0dff93
Merge pull request #120 from chpc-tech-eval/tut0/release
nyameko Jul 8, 2024
fa1fec0
CHPC 2024 Student Cluster Competition - Day 1
nyameko Jul 8, 2024
1e38018
Merge pull request #121 from chpc-tech-eval/tut0/release
nyameko Jul 8, 2024
0206950
Merge pull request #122 from chpc-tech-eval/stag
nyameko Jul 8, 2024
96b5c72
Updated TOC.
nyameko Jul 8, 2024
e74b6d4
Updated percentages.
nyameko Jul 8, 2024
6c98efe
Merge pull request #123 from chpc-tech-eval/tut0/release
nyameko Jul 8, 2024
1b07c0f
Updated TOC.
nyameko Jul 8, 2024
a6f712b
Updated percentages.
nyameko Jul 8, 2024
1ee1f1e
Incremental
nyameko Jul 8, 2024
64a9e9c
Fixed link to Dell Pricelist.
nyameko Jul 8, 2024
02f29a9
Minor change.
nyameko Jul 8, 2024
439db6f
Merge pull request #124 from chpc-tech-eval/tut0/release
nyameko Jul 8, 2024
77b2800
Fixed day 1 bugs, security groups and flavor names.
nyameko Jul 8, 2024
52c0b95
Merge pull request #125 from chpc-tech-eval/tut0/release
nyameko Jul 8, 2024
a051323
Merge branch 'main' into stag
nyameko Jul 8, 2024
f19accc
Merge pull request #126 from chpc-tech-eval/stag
nyameko Jul 8, 2024
76ebf12
Adding tut 2.
nyameko Jul 8, 2024
54041da
Tut 1 indenting.
nyameko Jul 8, 2024
bf81dcd
Fixed code indent.
nyameko Jul 8, 2024
679fb73
Added ZeroTier Section.
nyameko Jul 9, 2024
ec9c293
Added WIreGuard and TKA.
nyameko Jul 9, 2024
37beb28
Merge pull request #128 from chpc-tech-eval/tut0/release
nyameko Jul 9, 2024
63932d9
Merge branch 'main' into stag
nyameko Jul 9, 2024
3c8d8ee
Merge pull request #129 from chpc-tech-eval/stag
nyameko Jul 9, 2024
6c4fa77
Fixed links.
nyameko Jul 9, 2024
45bc398
Updated Intro, Table of Contents, Ansilble and Main landing page.
nyameko Jul 9, 2024
f3e7967
Incremental tut 2 updates.
nyameko Jul 9, 2024
a97b50a
fix some links to subsections
sfebruary Jul 9, 2024
ac62fb7
minor change: subsection to section
sfebruary Jul 9, 2024
09434c8
update checklist for the monitoring
sfebruary Jul 9, 2024
9f23b1e
minor wording update
sfebruary Jul 9, 2024
ff72642
Updated Ansible.
nyameko Jul 9, 2024
768f185
Updated TOC.
nyameko Jul 9, 2024
6423e9f
Updating TOC on Main Page.
nyameko Jul 9, 2024
55db706
Updated TOCs.
nyameko Jul 9, 2024
7c304a7
Merge branch 'stag' into tut0/release
nyameko Jul 9, 2024
493cb26
Merge pull request #133 from chpc-tech-eval/tut0/release
nyameko Jul 9, 2024
a3f8a7b
Merge branch 'main' into stag
nyameko Jul 9, 2024
4241bbd
Merge pull request #134 from chpc-tech-eval/stag
nyameko Jul 9, 2024
3f45262
Fixed tables, numbering and indenting.
nyameko Jul 9, 2024
3a3a3a4
Updated SSH Daemon.
nyameko Jul 9, 2024
875c050
Uploaded youtube video links.
nyameko Jul 9, 2024
8730d16
Merge pull request #135 from chpc-tech-eval/tut0/release
nyameko Jul 9, 2024
7903ed8
Merge branch 'main' into stag
nyameko Jul 9, 2024
9df88e2
Merge pull request #136 from chpc-tech-eval/stag
nyameko Jul 9, 2024
d8a5c62
Changed code block for sshing into compute.
nyameko Jul 9, 2024
44d23f1
Merge pull request #137 from chpc-tech-eval/tut0/release
nyameko Jul 9, 2024
ff22de3
Merge branch 'main' into stag
nyameko Jul 9, 2024
f18e4c8
Merge pull request #138 from chpc-tech-eval/stag
nyameko Jul 9, 2024
e4ddf8f
Fixed indenting in Creating Users.
nyameko Jul 9, 2024
9a72183
Fixed indenting on user creation.
nyameko Jul 9, 2024
87b9a8b
Fixed indenting.
nyameko Jul 9, 2024
04ce380
Fixed typo in Zama's name.
nyameko Jul 9, 2024
853c345
Updated ssh to compute node, force password auth for example.
nyameko Jul 9, 2024
f81f2fd
Merge pull request #139 from chpc-tech-eval/tut0/release
nyameko Jul 9, 2024
4c6d975
Merge pull request #140 from chpc-tech-eval/stag
nyameko Jul 9, 2024
75bd09c
Added tut 3 and cleared a number of issues.
nyameko Jul 9, 2024
35ee6ed
Added output of top.
nyameko Jul 9, 2024
2bfb5e1
Completed LAMMPS.
nyameko Jul 10, 2024
cea3541
Finished Benchmarking.
nyameko Jul 10, 2024
34181e4
Finished HPL.
nyameko Jul 10, 2024
3d82f98
Fixed TOCs.
nyameko Jul 10, 2024
fee3005
Merge pull request #143 from chpc-tech-eval/tut3/release
nyameko Jul 10, 2024
983bbb5
Fixed HPL details.
nyameko Jul 10, 2024
eea29a1
Merge pull request #144 from chpc-tech-eval/tut3/release
nyameko Jul 10, 2024
4f754ea
Fixed top500 table.
nyameko Jul 10, 2024
10d7941
Merge pull request #145 from chpc-tech-eval/tut3/release
nyameko Jul 10, 2024
b108b2d
Merge branch 'main' into stag
nyameko Jul 10, 2024
1b99bb4
Merge pull request #146 from chpc-tech-eval/stag
nyameko Jul 10, 2024
1a117ed
Fixed nftables permissions design issues.
nyameko Jul 10, 2024
848735b
Added `sudo` for exportsfs.
nyameko Jul 10, 2024
cc5ef70
Merge pull request #147 from chpc-tech-eval/tut3/release
nyameko Jul 10, 2024
9b410d4
Fixed SELinux.
nyameko Jul 10, 2024
49d329c
Merge branch 'stag' into tut3/release
nyameko Jul 10, 2024
1a2b6c4
Merge pull request #148 from chpc-tech-eval/tut3/release
nyameko Jul 10, 2024
a33525f
Merge branch 'main' into stag
nyameko Jul 10, 2024
8d169b9
Merge pull request #149 from chpc-tech-eval/stag
nyameko Jul 10, 2024
8fe4279
Merge pull request #132 from chpc-tech-eval/tut4/checklist
nyameko Jul 10, 2024
80313e6
Updated Ansible.
nyameko Jul 10, 2024
a48c39d
fixed ansible playbooks.
nyameko Jul 10, 2024
b2b0263
Merge pull request #150 from chpc-tech-eval/tut3/release
nyameko Jul 10, 2024
56b318e
Merge pull request #151 from chpc-tech-eval/stag
nyameko Jul 10, 2024
e2e1322
Updated wireguard instructions.
nyameko Jul 10, 2024
fbfe1eb
Merge branch 'stag' into tut3/release
nyameko Jul 10, 2024
e950092
Merge pull request #152 from chpc-tech-eval/tut3/release
nyameko Jul 10, 2024
a4cf373
Merge branch 'main' into stag
nyameko Jul 10, 2024
a37d029
Merge pull request #153 from chpc-tech-eval/stag
nyameko Jul 10, 2024
3a6fd52
fixing the monitoring subsection structure
sfebruary Jul 10, 2024
db4450a
small refactor of a particular monitoring subsection
sfebruary Jul 10, 2024
60fe9cd
further cleanups, and adding in a section for the legacy approach
sfebruary Jul 10, 2024
41f618e
instructions to add in compute nodes to the mix
sfebruary Jul 10, 2024
aa6b641
fix missing subsection wording
sfebruary Jul 10, 2024
6821cfe
Merge pull request #154 from chpc-tech-eval/tut4/monitoring_update
nyameko Jul 11, 2024
8d9dc78
Merging staging back into dev to integrate tutorial 4 changes.
nyameko Jul 11, 2024
432f678
Merge branch 'dev' into merge_stag_into_dev
nyameko Jul 11, 2024
70202ea
Merge pull request #157 from chpc-tech-eval/merge_stag_into_dev
nyameko Jul 11, 2024
db03261
Incremental updates to tut4, will release later today.
nyameko Jul 11, 2024
249d374
Emboldened Titles.
nyameko Jul 11, 2024
eec917f
Updated indenting and bulletting.
nyameko Jul 11, 2024
4ad83be
Updated TOC.
nyameko Jul 11, 2024
6a52c05
Added port forwarding description.
nyameko Jul 11, 2024
0892fbc
Added alternative entries for connecting to head node.
nyameko Jul 11, 2024
65199c9
Updated qv and terraform steps.
nyameko Jul 11, 2024
bd3d866
Updated TOC.
nyameko Jul 11, 2024
4b8a8c1
Incremental commits to expand CircleCI.
nyameko Jul 11, 2024
52d5814
Reorganized order of sections, easier sections first.
nyameko Jul 11, 2024
120b950
Wrapped up CircleCI, removed Ansible.
nyameko Jul 11, 2024
09d24f9
Updated TOC in main.
nyameko Jul 11, 2024
c974ba8
Added change management description.
nyameko Jul 11, 2024
eefef5a
Added main landing page description.
nyameko Jul 11, 2024
6bdc36c
Updated TOC and checklist.
nyameko Jul 11, 2024
4a18d84
Fixed tab indexing.
nyameko Jul 11, 2024
1ceba8b
Merge branch 'main' into tut4/release
nyameko Jul 11, 2024
fccdcb8
Merge pull request #159 from chpc-tech-eval/tut4/release
nyameko Jul 11, 2024
6fe1f43
Merge pull request #160 from chpc-tech-eval/tut4/release
nyameko Jul 11, 2024
eef0f51
Merge pull request #161 from chpc-tech-eval/tut4/release
nyameko Jul 11, 2024
bd07d6e
Added section for specifically building HPL on Ubuntu-based systems.
nyameko Nov 12, 2024
e1db78e
Merge branch 'stag' into dev
nyameko Nov 12, 2024
1db6bdb
Merge pull request #220 from chpc-tech-eval/dev
nyameko Nov 13, 2024
a407387
Merge branch 'main' into stag
nyameko Nov 13, 2024
010688c
Merge pull request #221 from chpc-tech-eval/stag
nyameko Nov 13, 2024
98e91c7
We have created a README.md and we documented everything related to a…
mohhlahlanewilliam Nov 23, 2024
ba669fc
Update README.md
mohhlahlanewilliam Nov 23, 2024
31de30b
Update README.md
mohhlahlanewilliam Nov 23, 2024
a71d09c
Update README.md
mohhlahlanewilliam Nov 23, 2024
03b4ef0
Updated the README.md file for tutorial 3 section 8 by writing the st…
mohhlahlanewilliam Nov 24, 2024
334046b
Merge branch 'aws' of https://github.com/mohhlahlanewilliam/ThunderBy…
mohhlahlanewilliam Nov 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Completed working on ntp and nfs.
nyameko committed Jul 7, 2024
commit 5e6f6c9d22f4b63c95e9f2a22e993006ca2f3294
121 changes: 30 additions & 91 deletions tutorial2/README.md
Original file line number Diff line number Diff line change
@@ -548,105 +548,44 @@ The head node will act as the [NFS server](https://docs.rockylinux.org/guides/fi


1. Install the NFS Utilities on both the head node and compute node(s):

```bash
sudo dnf install nfs-utils
```

5. NFS shares (directories on the NFS server) are configured in the `/etc/exports` file. Here you specify the directory you want to share, followed by the IP address or range you want to share to and then the options for sharing. We want to export the `/home` directory, so edit `/etc/exports` and add the following:

```conf
/home 10.100.50.0/24(rw,async,no_subtree_check,no_root_squash)
```


Let us go through all the options for NFS exports.
- rw - gives the client machine read and write access on the NFS volume.
- async - this option forces NFS to write changes to the disk before replying. This option is considered more reliable. However, it also reduces the speed of file operations.

- no_subtree_check - this option prevents subtree checking, a process where the host must check whether the file is available along with permissions for every request. It can also cause issues when a file is renamed on the host while still open on the client. Disabling it improves the reliability of NFS.

- no_root_squash - By default, NFS translates requests from a root user on the client into a non-privileged user on the host. This option disables that behavior and should be used carefully to allow the client to gain access to the host.


6. Start and enable the `nfs-server` service using `systemctl`.

```bash
sudo systemctl enable --now nfs-server rpcbind
```


7. NFS on Rocky Linux makes use of three different services, and they all need to be allowed through your firewall. You can add these rules with firewall-cmd:

```bash
sudo firewall-cmd --add-service={nfs,mountd,rpc-bind} --permanent
sudo firewall-cmd --reload
```

## NFS Client (Compute Node)

The compute node acts as the client for the NFS, which will mount the directory that was exported from the server (`/home`). Once mounted, the compute node will be able to interact with and modify files that exist on the head node and it will be synchronised between the two.

### Mounting An NFS Mount

The `nfs-utils, nfs4-acl-tools` packages need to be installed before you can do anything NFS related on the compute node.

```bash
sudo dnf install nfs-utils nfs4-acl-tools
```

Since the directory we want to mount is the `/home` directory, the user can not be in that directory.


1. Mount the /home directory from the head node using the `mount` command:

```bash
sudo mount -t nfs <headnode_ip_or_hostname>:/home /home
```

2. Once done, you can verify that the `/home` directory of the head node is mounted by using `df -h`:

```bash
df -h
```


With this mounted, it effectively replaces the `/home` directory of the compute node with the head node's one until it is unmounted. To verify this, create a file on the compute node's `rocky` user home directory (`/home/rocky`) and see if it is also automatically on the head node. If not, you may have done something wrong and may need to redo the above steps!

### Making The NFS Mount Permanent

Using `mount` from the command line will not make the mount permanent. It will not survive a reboot. To make it permanent, we need to edit the `/etc/fstab` file on the compute node. This file contains the mappings for each mount of any drives or network locations on boot of the operating system.

1. First we need to unmount the mount we made:

```bash
sudo umount /home
```

2. Now we need to edit the `/etc/fstab` file and add this new line to it (be careful not to modify the existing lines!):


```bash
headnode.cluster.scc:/home /home nfs auto,nofail,noatime,nolock,tcp,actimeo=1800,intr 0 0
```

The structure is: `<host>:<filesystem_dir> <local_location> <filesystem_type> <filesystem_options> 0 0`. The last two digits are not important for this competition and can be left at 0 0.
1. Edit `/etc/exports` on the head node

For the description of nfs options listed above see the link: https://cheatography.com/baa/cheat-sheets/fstab-nfs/#google_vignette
NFS shares (directories on the NFS server) are configured in the `/etc/exports` file. Here you specify the directory you want to share, followed by the IP address or range you want to share to and then the options for sharing. We want to export the `/home` directory, so edit `/etc/exports` and add the following:

3. With this done, we can mount the new `/etc/fstab` entry:

```bash
sudo mount -a
```

4. Once again, you can verify that the `/home` directory of the head node is mounted by using `df -h`:

```bash
df -h
```
```conf
/home 10.100.50.0/24(rw,async,no_subtree_check,no_root_squash)
```
* `rw` gives the client machine read and write access on the NFS volume.
* `async` forces NFS to write changes to the disk before replying. This option is considered more reliable. However, it also reduces the speed of file operations.
* `no_subtree_check` prevents a process where the host must check whether the file is available along with permissions for every request. It can also cause issues when a file is renamed on the host while still open on the client. Disabling it improves the reliability of NFS.
* `no_root_squash` disables the default behavior where NFS translates requests from a root user on the client, into a non-privileged user on the host. Great care should be taken when allowing the client to gain access to the host with this setting.

1. Open TCP port 2049 on your head node's firewall by editing `/etc/nftables/hn.nft`, and restarting the `nftables` service

1. Export the shares, then start and enable the `nfs-server` service using `systemctl` on the head node.
```bash
exportfs -ar
sudo systemctl enable nfs-server
```
1. Mount the NFS export on your compute node
```bash
# You cannot mount /home while you are occupying it
cd /
sudo mount -t nfs <headnode_ip>:/home /home
```
1. Verify that you successfully mounted `/home` export
```bash
df -h
```
1. Edit your `/etc/fstab` to make the effect persist after a restart. Add this entry to the end of your fstab:
```conf
<headnode_ip>:/home /home nfs defaults,timeo=1800,retrans=5,_netdev 0 0
```
servername:/music /mountpoint/on/client nfs defaults,timeo=900,retrans=5,_netdev 0 0

# Passwordless SSH
## Generating SSH Keys on your Head Node