Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

start adding small utilities for Slurm clusters #19

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions slurm/sshow
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#!/bin/bash

set -o errexit \
-o nounset \
-o pipefail

function check_command () {
local cmd="$1"
if ! command -v "$cmd" &>/dev/null; then
>&2 \
printf "Error: no ${cmd} found, cannot do anything"
exit 1
fi
}

function usage() {
>&2 \
printf "
%s - a wrapper around Slurm tools, to simplify common ops

Subcommands:

account - accounts with orgs and coordinators (sacctmgr)
assoc - associations with users (sacctmgr)
qos - qos details (sacctmgr)
user - usernames with their default account and admin status (sacctmgr)
node - nodes with their current state and load (sinfo)
job - current jobs (squeue)
jobhist - past jobs (sacct)

You can pass additional options to the underlying tools. Please refer to the
corresponding man pages.

" \
"$(basename "$0")"
}

if [[ "$#" -lt 1 ]]; then
usage
exit 1
fi

subcmd="$1"
shift

case "$subcmd" in
# These are all very simple at the moment. If they get complicated, split them out into a function.
"account")
check_command sacctmgr
exec sacctmgr show account \
format='account%-10,org%-8,coordinators%-9,descr%-60' \
"$@"
;;
"assoc")
check_command sacctmgr
exec sacctmgr show assoc \
format='account%-10,user%-8,qos%-60,share,maxsubmit' \
"$@"
;;
"qos")
check_command sacctmgr
exec sacctmgr show qos \
format='name%-10,priority%4,mintres%-12,maxtres%-12,flags%-30,maxjobsperuser,grpjobs,maxsubmit' \
"$@"
;;
"user")
check_command sacctmgr
exec sacctmgr show user \
format='user%-8,defaultaccount%-10,admin%-16' \
"$@"
;;
"node")
check_command sinfo
exec sinfo --Node \
--Format 'NodeHost:16,Available:6,CPUsState:14,CPUsLoad:.8,Memory:.8 ,FreeMem:.8 ,StateCompact:8,Reason:40' \
"$@"
;;
"jobhist")
check_command sacct
exec sacct --allocations --duplicates \
--format 'jobid%8,user%-8,account%-10,qos%-10,state%-10,start%19,elapsed%10,ncpus%5,exitcode,jobname%-24' \
"$@"
;;
"job")
check_command squeue
exec squeue \
--Format 'jobid:.8 ,username:8 ,account:10 ,qos:10 ,state:10 ,starttime:.19 ,timeused:.10 ,numcpus:.5 ,priority:.10 ,name:24' \
"$@"
;;
*)
# This implements something like git's handling of scripts named e.g. git-thing
if command -v "sshow-$subcmd"; then
exec "sshow-$subcmd" "$@"
else
printf "Error: unrecognised subcommand: %s\n" "$subcmd" >&2
exit 1
fi
;;
esac


Unchanged files with check annotations Beta

#!/usr/bin/env python3
import os.path

Check failure on line 3 in mmm/thomas/allocations.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/allocations.py:3:8: F401 `os.path` imported but unused

Check failure on line 3 in mmm/thomas/allocations.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/allocations.py:3:8: F401 `os.path` imported but unused
import sys
import configparser

Check failure on line 5 in mmm/thomas/allocations.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/allocations.py:5:8: F401 `configparser` imported but unused

Check failure on line 5 in mmm/thomas/allocations.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/allocations.py:5:8: F401 `configparser` imported but unused
import argparse
#import csv
import numpy

Check failure on line 8 in mmm/thomas/allocations.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/allocations.py:8:8: F401 `numpy` imported but unused

Check failure on line 8 in mmm/thomas/allocations.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/allocations.py:8:8: F401 `numpy` imported but unused
import pandas
def getargs(argv):
print("Warning [Person]: Detected unknown key: " + a + ": " + str(PersonDict[a]))
self.Title=PersonDict["Name"]["Title"]
if type(self.Title) == type(None):

Check failure on line 207 in mmm/thomas/safe_json_decoder.py

GitHub Actions / Ruff-Py

Ruff (E721)

mmm/thomas/safe_json_decoder.py:207:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks

Check failure on line 207 in mmm/thomas/safe_json_decoder.py

GitHub Actions / Ruff-Py

Ruff (E721)

mmm/thomas/safe_json_decoder.py:207:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
self.Title = ""
self.FirstName=PersonDict["Name"]["Firstname"]
self.LastName=PersonDict["Name"]["Lastname"]
# Convert JSON data structure to a list of objects
def JSONDataToTickets(JSONData):
Tickets = []
if type(JSONData) == list:

Check failure on line 247 in mmm/thomas/safe_json_decoder.py

GitHub Actions / Ruff-Py

Ruff (E721)

mmm/thomas/safe_json_decoder.py:247:8: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks

Check failure on line 247 in mmm/thomas/safe_json_decoder.py

GitHub Actions / Ruff-Py

Ruff (E721)

mmm/thomas/safe_json_decoder.py:247:8: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
for a in JSONData:
Tickets.append(AccountRequest(a))
else:
# If this is run directly, process test.json in the current working directory
# and print the output as a string.
if __name__=="__main__":
import json

Check failure on line 257 in mmm/thomas/safe_json_decoder.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_json_decoder.py:257:12: F401 `json` imported but unused

Check failure on line 257 in mmm/thomas/safe_json_decoder.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_json_decoder.py:257:12: F401 `json` imported but unused
import sys
filename="test.json"
import sys
import configparser
import argparse
import subprocess

Check failure on line 7 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_tickets.py:7:8: F401 `subprocess` imported but unused

Check failure on line 7 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_tickets.py:7:8: F401 `subprocess` imported but unused
import validate

Check failure on line 8 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_tickets.py:8:8: F401 `validate` imported but unused

Check failure on line 8 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_tickets.py:8:8: F401 `validate` imported but unused
import mysql.connector
from mysql.connector import errorcode
from tabulate import tabulate

Check failure on line 11 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_tickets.py:11:22: F401 `tabulate.tabulate` imported but unused

Check failure on line 11 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F401)

mmm/thomas/safe_tickets.py:11:22: F401 `tabulate.tabulate` imported but unused
import json
import requests
import safe_json_decoder as decoder
try:
data = request.json()
return data
except json.decoder.JSONDecodeError as err:

Check failure on line 54 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F841)

mmm/thomas/safe_tickets.py:54:48: F841 Local variable `err` is assigned to but never used

Check failure on line 54 in mmm/thomas/safe_tickets.py

GitHub Actions / Ruff-Py

Ruff (F841)

mmm/thomas/safe_tickets.py:54:48: F841 Local variable `err` is assigned to but never used
print("Received invalid json, contents: " + str(request.content))
exit(1)
else: