Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for German DNS provider Core Networks #494

Open
wants to merge 98 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
0958442
Initial provider for Core Networks
MasinAD Apr 12, 2020
e207e99
Fixed corenetwork:Provider._update_record
MasinAD Apr 24, 2020
e6f383e
Compatibility fix for Python 2.7
MasinAD Apr 24, 2020
dd0ba44
Recording missed a call
MasinAD Apr 24, 2020
23878ca
Changed Exception message to what Certbot's dns_common_lexicon.py exp…
MasinAD Apr 26, 2020
e10f6c2
Merge branch 'master' into pr/494
adferrand Jun 1, 2020
98821c1
integrated Logging
MasinAD Jan 7, 2021
b1ab75d
Merge pull request #1 from AnalogJ/master
MasinAD Jan 7, 2021
5bc660d
removed some logging messages, added logic for failing gracefully in …
MasinAD Jan 8, 2021
2b8bb14
Added some clarification to the auth-file parameter help.
MasinAD Jan 8, 2021
421460d
Removed commented out code in provider test.
MasinAD Jan 8, 2021
52977c0
Switched to contextmanager for committing changes
MasinAD Jan 8, 2021
24c5a8f
Merge branch 'master' into pr/494
adferrand Aug 9, 2021
b05a8fd
Relax dependencies
adferrand Oct 3, 2021
198385c
Update dependencies and fix doc
adferrand Oct 3, 2021
8d2853a
Changelog
adferrand Oct 3, 2021
81080b4
Version 3.7.1
adferrand Oct 3, 2021
3ae7d45
Troubleshoot issues with Azure (#971)
adferrand Oct 4, 2021
58f8990
Add changelog
adferrand Oct 4, 2021
c0d4221
Version 3.8.0
adferrand Oct 4, 2021
0862b17
Update dependencies
adferrand Oct 12, 2021
8f90a01
Fix invalid API request to Rackspace Cloud DNS - Issue #981 (#989)
mattgauf Oct 15, 2021
fae4b77
Prepare changelog
adferrand Oct 15, 2021
243c008
Version 3.8.1
adferrand Oct 15, 2021
48e8421
Update dreamhost _authenticate to use dns-list_records instead of dom…
ryan953 Nov 3, 2021
3f4dfca
Prepare changelog
adferrand Nov 3, 2021
1765243
Version 3.8.2
adferrand Nov 3, 2021
e965bee
Update dependencies
adferrand Nov 3, 2021
531c37c
Update dependencies
adferrand Nov 11, 2021
aac90a1
Update windows environments
adferrand Nov 11, 2021
4214b64
Update dependencies and changelog
adferrand Nov 12, 2021
ce25e25
Fix find_site, dataset is not a valid child of filter, but a child of…
spike77453 Nov 11, 2021
cb87590
🐛 update Namecheap's nameserver list (#911)
dudeofawesome Nov 11, 2021
a1c414f
Version 3.8.3
adferrand Nov 12, 2021
8b72bf5
Update CHANGELOG.md
adferrand Nov 12, 2021
2b3a0ec
Update of the netcup nameserver (#1016)
mustermann2021 Nov 14, 2021
e1ccb12
Update Nameserver for INWX (#1015)
mustermann2021 Nov 14, 2021
b5daac1
Support & test Python 3.10 (#1017)
adferrand Nov 14, 2021
f65991e
Update dependencies
adferrand Nov 18, 2021
bd1905f
Take description of the provider into account in the doc
adferrand Nov 24, 2021
d8e2b4d
Fix return
adferrand Nov 24, 2021
fd8c361
Ensure TTL is provided as an integer (#1031)
parthjoshi-pc Nov 25, 2021
23c521c
Update dependencies
adferrand Dec 28, 2021
4719982
Fix tox config
adferrand Dec 28, 2021
c8d15b7
Fix tox deps
adferrand Dec 28, 2021
734ff83
Short fix for godaddy
adferrand Dec 28, 2021
39fe561
Fix lint
adferrand Dec 28, 2021
2753694
Add support for value-domain.com (#1018)
Dec 28, 2021
bb97f03
Prepare changelog
adferrand Dec 28, 2021
3bd9213
Update documentation
adferrand Dec 28, 2021
26ba265
Version 3.8.4
adferrand Dec 28, 2021
06c04ac
Complete redesign of the update logic in godaddy provider
adferrand Dec 29, 2021
1dce378
Prepare changelog
adferrand Dec 29, 2021
7cb82ad
Version 3.8.5
adferrand Dec 29, 2021
c508037
Factor common logic in godaddy
adferrand Dec 29, 2021
04f233f
Rewrite delete logic in godaddy provider
adferrand Dec 29, 2021
696309b
Version 3.8.6
adferrand Dec 29, 2021
e6ddbe1
Revert failed versions
adferrand Dec 29, 2021
d117b77
Version 3.8.5
adferrand Dec 29, 2021
c0694ff
Drop Python 3.6 support
adferrand Jan 6, 2022
c2b409f
Update dependencies
adferrand Jan 6, 2022
c1b6078
Version 3.9.0
adferrand Jan 6, 2022
6b96a7a
added Porkbun (#1062)
juanalbglz Jan 9, 2022
b1748e2
Update dependencies
adferrand Jan 9, 2022
2c5f784
fix conoha provider (specify auth_region when get provider option) (#…
k-serenade Jan 9, 2022
65cb705
Exclude integration tests from source distributions + remove MANIFEST…
sbraz Jan 9, 2022
509223b
Use known domain as filter (#954)
dgrothaus-ku Jan 9, 2022
443e2cb
Prepare build of version doc
adferrand Jan 17, 2022
3c4b376
Reimplement transip provider with the API REST v6 (#1086)
adferrand Jan 17, 2022
1be1d8d
Prepare changelog
adferrand Jan 17, 2022
137527f
Version 3.9.1
adferrand Jan 17, 2022
ac8a57a
Update documentation
adferrand Jan 17, 2022
7a8470a
Reorganize documentation
adferrand Jan 17, 2022
1b1372c
Prepare changelog
adferrand Jan 17, 2022
6e82b5e
Version 3.9.2
adferrand Jan 17, 2022
32a6fd5
Use appropriate JSONDecodeError for retrocompatibility purposes (#1100)
adferrand Jan 26, 2022
1e18d4e
Prepare changelog
adferrand Jan 26, 2022
877afc0
Version 3.9.3
adferrand Jan 26, 2022
e7f6e60
Bump boto3 from 1.20.44 to 1.20.53 (#1124)
dependabot[bot] Feb 13, 2022
3092ad0
Bump types-toml from 0.10.3 to 0.10.4 (#1123)
dependabot[bot] Feb 13, 2022
766561b
Bump types-requests from 2.27.7 to 2.27.9 (#1121)
dependabot[bot] Feb 13, 2022
6fbe201
Bump oci from 2.55.0 to 2.56.0 (#1120)
dependabot[bot] Feb 13, 2022
99b3c2c
Bump types-setuptools from 57.4.7 to 57.4.9 (#1116)
dependabot[bot] Feb 13, 2022
f84c1bd
Bump softlayer from 5.9.8 to 5.9.9 (#1115)
dependabot[bot] Feb 13, 2022
78a8fd6
Bump pytest from 6.2.5 to 7.0.0 (#1114)
dependabot[bot] Feb 13, 2022
4db0a0f
Updated valid record types for DreamHost integration (#1110)
JerrettDavis Feb 13, 2022
6ca8dbe
Bump black from 21.12b0 to 22.1.0 (#1107)
dependabot[bot] Feb 13, 2022
fb5004e
Bump types-pyyaml from 6.0.3 to 6.0.4 (#1105)
dependabot[bot] Feb 13, 2022
bb533ff
Bump dnspython from 2.1.0 to 2.2.0 (#1089)
dependabot[bot] Feb 13, 2022
f78cd4f
Update dependencies
adferrand Feb 13, 2022
1c10cc5
Add Webgo Provider (#1102)
mod242 Feb 13, 2022
e969090
Update documentation
adferrand Feb 13, 2022
58508b3
Prepare changelog
adferrand Feb 14, 2022
61e26a6
Version 3.9.4
adferrand Feb 14, 2022
314ab08
Merge branch 'master' of https://github.com/AnalogJ/lexicon
MasinAD Mar 29, 2022
219455d
removed credentials
MasinAD Mar 29, 2022
02c0aac
Fixed corenetworks provider not authenticating in tests
MasinAD Mar 30, 2022
06d2c30
Merge branch 'master' into pr/494
adferrand Apr 19, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
integrated Logging
MasinAD committed Jan 7, 2021
commit 98821c17613644e070aeec1d838bcb33e7c41902
68 changes: 42 additions & 26 deletions lexicon/providers/corenetworks.py
Original file line number Diff line number Diff line change
@@ -1,41 +1,50 @@
"""Module provider for DNS Simple"""
"""Module provider for Core Networks"""
from __future__ import absolute_import
import json
import hashlib
import logging
import time
import os
import tempfile

import requests
from lexicon.providers.base import Provider as BaseProvider

LOGGER = logging.getLogger(__name__)
CorenetworksLog = logging.getLogger(__name__)
#CorenetworksLog.setLevel(logging.DEBUG)
CorenetworksLog.info("Starting %s" % __name__)

NAMESERVER_DOMAINS = ['core-networks.de', 'core-networks.eu', 'core-networks.com']

def provider_parser(subparser):
"""Configure provider parser for Core Networks"""
subparser.add_argument(
"--auth-username", help="specify login for authentication")
"--auth-username", help="Specify login for authentication")
subparser.add_argument(
"--auth-password", help="specify password for authentication")
"--auth-password", help="Specify password for authentication")
subparser.add_argument(
"--auth-file", help="specify location for authentication file")
"--auth-file", help="Specify location for authentication file. If this contains a valid token it will be used. Otherwise --auth-username and --auth-password are necessary.")

class Provider(BaseProvider):
"""Provider class for Core Networks"""
def __init__(self, config):
CorenetworksLog.info("Initialising class Provider")
super(Provider, self).__init__(config)
self.domain_id = None
self.account_id = None
self.token = None
self.expiry = None
self.modified = False
self.auth_file = { 'token': None, 'expiry': None }
CorenetworksLog.info("Auth file parameter: %s" % self._get_provider_option('auth_file'))
# Core Networks enforces a limit on the amount of logins per minute.
# As the token is valid for 1 hour it's sensible to store it for
# later usage.
self.auth_file_path = self._get_provider_option('auth_file') or '/tmp/corenetworks_auth.json'
if os.path.exists(os.path.expanduser("~")) and os.path.expanduser("~") != '':
path = os.path.expanduser("~")
else:
path = tempfile.gettempdir()
self.auth_file_path = self._get_provider_option('auth_file') or (path+'/corenetworks_auth.json')
self.api_endpoint = 'https://beta.api.core-networks.de'

def __del__(self):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your comment on the PR about slow operations. Which delay are we talking about ?

In theory, when Lexicon returns from the current operation one would assume that changes are committed. So for the sake of simplicity I would just commit right after the actual modification, and so hang until it is done.

If it is really a big problem, we can find a better way than calling the constructor. Using atexit standard module, you can register handlers that will be invoked when Python close the process. It gives a consistent way to ensure that it is executed, since the only way to avoid these handlers is to kill -9 the current process.

Still the Python process will hang while leaving until the commit is effective.

Copy link
Collaborator

@adferrand adferrand May 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it again. So if the slow operation is still acceptable to be done after each effective modification (create, update, delete).

If you want a pythonic approach to ensure that something is always done at the end of some logic, you can use a contextmanager. It is basically a try/finally, but more elegant. Here an implementation I propose for your use case:

from contextlib import contextmanager


class _CommitTrigger:
    def __init__(self):
        self._need_commit = False

    @property
    def need_commit(self):
        return self._need_commit

    @need_commit.setter
    def need_commit(self, set_need_commit):
        self._need_commit = set_need_commit


class Provider(BaseProvider):
    @contextmanager
    def ensure_commit(self):
        commit_trigger = _CommitTrigger()
        try:
            yield commit_trigger
        finally:
            if commit_trigger.need_commit:
                self._post("/dnszones/{0}/records/commit".format(self.domain))

    def _list_records(self, rtype=None, name=None, content=None):
        with self.ensure_commit() as commit_trigger:
            # Do logic ...

    def _create_record(self, rtype, name, content):
        with self.ensure_commit() as commit_trigger:
            # Do logic ...
            commit_trigger.need_commit = True
            
    # Add logic ...

Here the contextmanager will ensure that commit is done whatever happened in _create_record as soon as commit_trigger.need_commit = True is called. Similarly, list_record will not commit at the end, since commit_trigger.need_commit is not set to True.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your comment on the PR about slow operations. Which delay are we talking about ?

I don't know. When asking the provider for the limits they just mentioned the login limit. The commit limit seems to be different: The API responds with a variable delay; the more often the commit function is called the slower it gets. For normal operations it's almost unnoticeable but when deleting a lot of records e.g. after running the integration tests it takes much more time to commit each single delete than to commit all deletions at once.

I will take a closer look at your proposal. As I'm new to Python I'll need a little bit more time to understand what's happening there … 😃

I already worked in all other comments. I guess I'll be done tomorrow evening CEST.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adferrand: Okay, finally got to this point. We have decorators and contextmanager, with and yield statements, generators and iterables. Just let me say that I can wrap my head around the last two but with, yield and decorators are still beyond my scope 😆

I try to slowly understand what's happening here but that's too much at once. I guess I have to live with that for now.

With those changes I run again into this error in two tests:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://beta.api.core-networks.de/dnszones/fluchtkapsel.de/records/commit

The rest works as intended.

@@ -46,9 +55,13 @@ def __del__(self):
self.modified == False
return True

# Ref: https://beta.api.core-networks.de/doc/#functon_auth_token
def _authenticate(self):
self._log("Domain %s" % self.domain)
"""Authenticate by either providing stored access token or
acquiring and storing token. This method will query the
list of zones and store them for later use. If the requested
domain is not in the list of zones it will raise an exception.
Ref: https://beta.api.core-networks.de/doc/#functon_auth_token"""
CorenetworksLog.debug("Entering _authenticate, requesting domain %s" % self.domain)
self._refresh_auth_file()
if 'token' in self.auth_file:
self.token = self.auth_file['token']
@@ -79,6 +92,7 @@ def _list_records(self, rtype=None, name=None, content=None):
type, name and content are used to filter records.
If possible filter during the query, otherwise filter after response is received.
Ref: https://beta.api.core-networks.de/doc/#functon_dnszones_records"""
CorenetworksLog.debug("Entering _list_records")
zone = next((zone for zone in self.zones if zone["name"] == self.domain), None)
if not zone:
raise Exception('Domain not found')
@@ -100,6 +114,7 @@ def _list_records(self, rtype=None, name=None, content=None):

def _create_record(self, rtype, name, content):
"""Creates a record. If record already exists with the same content, do nothing."""
CorenetworksLog.debug("Entering _create_record")

# Check for existence of record.
existing_records = self._list_records(rtype, name, content)
@@ -116,6 +131,9 @@ def _create_record(self, rtype, name, content):
}
if self._get_lexicon_option('ttl'):
data['ttl'] = self._get_lexicon_option('ttl')
# Bug reported by chkpnt. If ttl is less than 60s the API throws a "415 Client Error: Unsupported Media Type"
if data['ttl'] < 60:
data['ttl'] = 60
if self._get_lexicon_option('priority'):
data['priority'] = self._get_lexicon_option('priority')

@@ -130,6 +148,7 @@ def _update_record(self, identifier, rtype=None, name=None, content=None):
"""Updates a record. Core Networks neither supports updating a record nor is able to reliably identify a record
after a change. The best we can do is to identify the record by ourselves, fetch its data, delete it and
re-create it."""
CorenetworksLog.debug("Entering _update_record")
if identifier is not None:
# Check for existence of record
existing_records = self._list_records(rtype)
@@ -148,8 +167,10 @@ def _update_record(self, identifier, rtype=None, name=None, content=None):
else:
records = self._list_records( rtype=rtype, name=self._relative_name(name) )
if len(records) > 0:
for record in records:
return self._update_record( record['id'], rtype, name, content )
if len(records) > 1:
CorenetworksLog.warning("Found %s records, will only update the first record in search result list." % len(records))
record = records[0]
return self._update_record( record['id'], rtype, name, content )
else:
return True
return False
@@ -158,6 +179,7 @@ def _delete_record(self, identifier=None, rtype=None, name=None, content=None):
"""Delete an existing record.
If record does not exist, do nothing.
Ref: https://beta.api.core-networks.de/doc/#functon_dnszones_records_delete"""
CorenetworksLog.debug("Entering _delete_record")
if identifier is not None:
# Check for existence of record
existing_records = self._list_records( rtype, name, content )
@@ -188,12 +210,13 @@ def _delete_record(self, identifier=None, rtype=None, name=None, content=None):
# Helpers

def _request(self, action='GET', url='/', data=None, query_params=None):
CorenetworksLog.debug("Entering _request")
if data is None:
data = {}
if query_params is None:
query_params = {}

self._log( "url: %s with data %s and query_params %s" % ( url, str(data), str(query_params) ) )
CorenetworksLog.debug( "url: %s with data %s and query_params %s" % ( url, str(data), str(query_params) ) )
default_headers = {}

if self.token:
@@ -212,8 +235,10 @@ def _request(self, action='GET', url='/', data=None, query_params=None):

return response.json() if response.text else None

# Ref: https://beta.api.core-networks.de/doc/#functon_dnszones
def _list_zones(self):
"""List existing zones.
Ref: https://beta.api.core-networks.de/doc/#functon_dnszones"""
CorenetworksLog.debug("Entering _list_zones")
return self._get('/dnszones/')

def _make_identifier(self, rtype, name, content):
@@ -231,7 +256,7 @@ def _refresh_auth_file(self):
auth.close()
return False
except IOError as e:
LOGGER.debug("No stored authentication found: %s. Acquiring token via API call." % os.strerror(e.errno))
CorenetworksLog.debug("No stored authentication found: %s. Acquiring token via API call." % os.strerror(e.errno))
self._get_token()
return True

@@ -247,20 +272,20 @@ def _commit_auth_file(self):
else:
return False
except IOError as e:
LOGGER.debug("Could not write authentication file: %s" % os.strerror(e.errno))
CorenetworksLog.debug("Could not write authentication file: %s" % os.strerror(e.errno))
finally:
auth.close()

def _get_token(self):
"""Request new token via API call"""
LOGGER.debug("Entering _get_token.")
CorenetworksLog.debug("Entering _get_token.")
data = {
'login' : self._get_provider_option('auth_username'),
'password': self._get_provider_option('auth_password')
}
self._log(str(data))
CorenetworksLog.debug(str(data))
payload = self._post('/auth/token', data = data)
LOGGER.debug("%s", str(payload))
CorenetworksLog.debug("%s", str(payload))
self.token = payload['token']
self.expiry = payload['expires'] + time.time()

@@ -269,12 +294,3 @@ def _get_token(self):
self.auth_file['expiry'] = self.expiry
self._commit_auth_file()

def _log(self, message):
l = open('/tmp/corenetworks.log', 'a+')
l.write(message+"\n")
l.close()

# def _get_provider_option(self, option):
# opt = self.config.resolve('lexicon:{0}:{1}'.format(self.provider_name, option))
# self._log("corenetworks: lexicon:%s:%s = %s" % (self.provider_name, option, opt) )
# return opt
1 change: 1 addition & 0 deletions lexicon/tests/providers/test_corenetworks.py
Original file line number Diff line number Diff line change
@@ -36,4 +36,5 @@ def _test_parameters_overrides(self):
'api_endpoint': 'https://beta.api.core-networks.de',
# 'auth_username': 'apiaccount',
# 'auth_password': 'apipassword'
'auth_file': ''
}