tobymao · tobymao · Aug 31, 2023 · Aug 30, 2023 · Aug 31, 2023 · Aug 31, 2023
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,62 @@
+# Changelog
+
+## Next
+- `saq` command-line client includes current path to simplify development
+- Added embeddable `saq.web.starlette`
+- Documentation now hosted on [Read The Docs](https://saq-py.readthedocs.io/)
+
+## v0.10
+- Always use FIFO queue behavior
+
+## v0.9
+
+### v0.9.4
+- Update redis requirements
+
+### v0.9.3
+- Fix timeout of 0
+- Allow `apply()` to take a negative ttl
+- Clean up signal handlers after shutdown event is triggered
+
+### v0.9.2
+- Job context is more dynamic
+- Limit redis to prevent breakage
+
+### v0.9.1
+- Fix worker queue settings
+- Better type info for `job_keys`
+- Add type annotations
+- Fixes ResourceWarning in tests
+
+### v0.9.0
+- basic auth fixes
+
+## v0.8
+
+### v0.8.1
+- Replace exception logger function with `logger.exception`
+- Correct param name in docstring for `Worker`
+- Documentation fixes
+
+### v0.8.0
+- Add `--quiet` to command-line client
+- Allow users to expire, keep, or disable storage of job information
+
+## v0.7
+
+### v0.7.1
+- Fix bug with queue map timeouts
+
+### v0.7.0
+- More robust error handling
+
+## v0.6
+
+### v0.6.2
+- Update readme
+
+### v0.6.1
+- Make upkeep more robust to errors
+
+### v0.6.0
+- Use queue name in the job key
diff --git a/Makefile b/Makefile
@@ -5,12 +5,13 @@ help:
 	@echo  "tobymao/saq dev makefile"
 	@echo  ""
 	@echo  "usage: make <target>"
-	@echo  " up	Force updates dev/test dependencies - attempts clean-install"
-	@echo  " deps	Ensure dev/test dependencies are installed"
-	@echo  " test	Runs all tests"
-	@echo  " lint	Reports all linter violations"
-	@echo  " ci	Runs lints & tests (as a full CI run would)"
-	@echo  " format Tries to auto-fix simpler linting issues"
+	@echo  " up		Force updates dev/test dependencies - attempts clean-install"
+	@echo  " deps		Ensure dev/test dependencies are installed"
+	@echo  " test		Runs all tests"
+	@echo  " lint		Reports all linter violations"
+	@echo  " ci		Runs lints & tests (as a full CI run would)"
+	@echo  " format		Tries to auto-fix simpler linting issues"
+	@echo  " devdocs	Builds docs and hosts them on port 8000"
 
 up:
 	pip freeze | grep -v "^-e" | xargs pip uninstall -y
@@ -37,8 +38,5 @@ format:
 
 ci: deps lint test
 
-run:
-	./etc/devrun.sh
-
 devdocs: deps_docs
 	sphinx-autobuild docs docs/_build/html
diff --git a/README.md b/README.md
@@ -1,6 +1,8 @@
 # SAQ
 SAQ (Simple Async Queue) is a simple and performant job queueing framework built on top of asyncio and redis. It can be used for processing background jobs with workers. For example, you could use SAQ to schedule emails, execute long queries, or do expensive data analysis.
 
+[Documentation](https://saq-py.readthedocs.io/)
+
 It uses [redis-py](https://github.com/redis/redis-py) >= 4.2.
 
 It is similar to [RQ](https://github.com/rq/rq) and heavily inspired by [ARQ](https://github.com/samuelcolvin/arq). Unlike RQ, it is async and thus [significantly faster](benchmarks) if your jobs are async. Even if they are not, SAQ is still considerably faster due to lower overhead.

diff --git a/docs/benchmarks.md b/docs/benchmarks.md
diff --git a/docs/changelog.md b/docs/changelog.md
@@ -0,0 +1,2 @@
+```{include} ../CHANGELOG.md
+```
diff --git a/docs/comparison.md b/docs/comparison.md
@@ -14,3 +14,21 @@ SAQ is heavily inspired by [ARQ](https://github.com/samuelcolvin/arq) but has se
     3. Handling of cancelled jobs different from failed jobs (machine redeployments)
 5. Before and after job hooks
 6. Easily run multiple workers to leverage more cores
+
+(benchmarks)=
+## Benchmarks
+
+```nasm
+pip install arq && python benchmarks/simple.py arq
+pip install saq && python benchmarks/simple.py saq
+pip install rq && python benchmarks/simple.py rq
+```
+
+### Results
+N = 1000, Results in seconds
+
+| Workflow |      saq | [arq](https://github.com/samuelcolvin/arq) | [rq](https://github.com/rq/rq) |
+|:---------|---------:|-------------------------------------------:|-------------------------------:|
+| enqueue  |  0.17970 |                                    0.54293 |                        0.48645 |
+| noop     |  0.91998 |                                    5.00900 |                        14.9416 |
+| sleep    |  2.82186 |                                    5.01430 |                        58.9042 |
diff --git a/docs/conf.py b/docs/conf.py
@@ -10,18 +10,33 @@
 #
 import datetime
 import os
+import re
 import sys
-sys.path.insert(0, os.path.abspath('../..'))
+
+sys.path.insert(0, os.path.abspath('.'))
+sys.path.insert(0, os.path.abspath('..'))
 
 
 # -- Project information -----------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
+def get_version():
+    verstrline = open("../saq/__init__.py", "rt").read()
+    mob = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]", verstrline, re.M)
+    if mob:
+        return mob.group(1)
+    else:
+        raise RuntimeError("Unable to find version string")
+
 project = 'SAQ'
 year = datetime.datetime.now().year
 copyright = f'{year}, Toby Mao'
 author = 'Toby Mao'
 
+# The short X.Y version
+version = get_version()
+# The full version, including alpha/beta/rc tags
+release = version
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
@@ -31,28 +46,41 @@
     'sphinx.ext.viewcode',
     'sphinx.ext.intersphinx',
     'myst_parser',
-    'autodoc2',
+    'autoapi.extension',
+    'sphinx_design',
 ]
 
 templates_path = ['_templates']
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 
 intersphinx_mapping = {
     'python': ('https://docs.python.org/3', None),
+    'redis': ('https://redis-py.readthedocs.io/en/stable/', None),
+    'typing_extensions': ('https://typing-extensions.readthedocs.io/en/latest/', None),
 }
 
-autodoc2_packages = [
-    {
-        "path": "../saq",
-        "auto_mode": True,
-        "exclude_files": [
-            "__main__.py"
-        ]
-    },
+autoapi_dirs = ['../saq']
+autoapi_ignore = [
+    '*/saq/__main__.py',
+    '*/saq/web/common.py'
+]
+autoapi_options = [
+    'members',
+    'undoc-members',
+    'private-members',
+    'show-inheritance',
+    'show-module-summary',
+    'special-members',
+    'imported-members',
 ]
+napoleon_use_admonition_for_notes = True
+napoleon_preprocess_types = True
+napoleon_attr_annotations = True
+highlight_language = 'python'
 
 myst_enable_extensions = [
-    "fieldlist"
+    "fieldlist",
+    "colon_fence",
 ]
 
 # -- Options for HTML output -------------------------------------------------

diff --git a/docs/contribute.md b/docs/contribute.md
@@ -0,0 +1,26 @@
+# Contribution
+
+## Development
+```sh
+python -m venv env
+source env/bin/activate
+pip install -e ".[dev,web]"
+docker run -p 6379:6379 redis
+./run_checks.sh
+```
+
+## Makefile
+SAQ has a `Makefile` which is used to simplify developing on the library:
+
+```text
+tobymao/saq dev makefile
+
+usage: make <target>
+ up             Force updates dev/test dependencies - attempts clean-install
+ deps           Ensure dev/test dependencies are installed
+ test           Runs all tests
+ lint           Reports all linter violations
+ ci             Runs lints & tests (as a full CI run would)
+ format         Tries to auto-fix simpler linting issues
+ devdocs        Builds docs and hosts them on port 8000
+```
diff --git a/docs/getting_started.md b/docs/getting_started.md
@@ -2,51 +2,19 @@
 
 ## Install
 
-### Minimal install
+::::{tab-set}
+:::{tab-item} Minimal install
 ```nasm
 pip install saq
 ```
+:::
 
-### Web + hiredis
+:::{tab-item} With Web UI & hiredis
 ```nasm
 pip install saq[web,hiredis]
 ```
-
-
-## Usage
-```text
-usage: saq [-h] [--workers WORKERS] [--verbose] [--web]
-           [--extra-web-settings EXTRA_WEB_SETTINGS]
-           [--port PORT] [--check]
-           settings
-
-Start Simple Async Queue Worker
-
-positional arguments:
-  settings              Namespaced variable containing
-                        worker settings eg: eg
-                        module_a.settings
-
-options:
-  -h, --help            show this help message and exit
-  --workers WORKERS     Number of worker processes
-  --verbose, -v         Logging level: 0: ERROR, 1: INFO,
-                        2: DEBUG
-  --web                 Start web app. By default, this
-                        only monitors the current
-                        worker's queue. To monitor
-                        multiple queues, see '--extra-
-                        web-settings'
-  --extra-web-settings EXTRA_WEB_SETTINGS, -e EXTRA_WEB_SETTINGS
-                        Additional worker settings to
-                        monitor in the web app
-  --port PORT           Web app port, defaults to 8080
-  --check               Perform a health check
-
-environment variables:
-  AUTH_USER     basic auth user, defaults to admin
-  AUTH_PASSWORD basic auth password, if not specified, no auth will be used
-```
+:::
+::::
 
 ## Example
 ```python
@@ -115,24 +83,13 @@ await queue.enqueue("test", a=1, scheduled=time.time() + 10)
 
 ## Demo
 
-Start the worker
-
+**Start the worker:**
 ```nasm
 saq examples.simple.settings --web
 ```
+Navigate to the Web UI: [http://localhost:8080](http://localhost:8080)
 
-Navigate to the [web ui](http://localhost:8080])
-
-Enqueue jobs
+**Enqueue jobs:**
 ```nasm
 python examples/simple.py
 ```
-
-## Development
-```
-python -m venv env
-source env/bin/activate
-pip install -e ".[dev,web]"
-docker run -p 6379:6379 redis
-./run_checks.sh
-```