Merge branch 'release/v0.16.0' #128

.github/PULL_REQUEST_TEMPLATE/bug-pull-request.md

@@ -0,0 +1,42 @@
+---
+name: Bug
+about: Provide specifics about a PR for a bug fix
+title: 'PR/Bug: ____(provide a descriptive name)'
+labels: bug, p1
+assignees: ''
+
+---
+<!-- IMPORTANT: Please do not create a PR without creating an issue first. -->
+
+<!-- Fields in **bold** are REQUIRED, fields in *italics* are OPTIONAL. -->
+
+**Description**
+<!-- A description of how this PR resolved the specified bug-->
+
+<!-- Add any linked issue(s) -->
+Fixes #
+
+
+**Update Changelog**
+<!-- Be brief, use imperative mood or simple noun phrases and add linked issues -->
+<!-- Examples: Improve verbosity of log messages #103 | GitHub actions for CI #105 -->
+
+- [ ] README.md, [changelog](../../README.md#changelog) <!-- update changelog here -->
+
+
+**Bump version**
+
+v0.xx.x -> v0.xx.x
+
+- [ ] README.md, [installation instructions](../../README.md#installation-instructions)
+- [ ] [`setup.py`](../../setup.py)
+- [ ] [`requiam/__init__.py`](../../requiam/__init__.py)
+
+
+*Screenshots or additional context*
+<!-- Add any other context about the problem here and/or screenshots to help explain the problem. -->
+
+
+*Testing (if applicable)*
+<!-- Explain how you tested this bug fix so that others can replicate it. -->
+<!-- Example: The exact commands you ran and their output. -->

.github/PULL_REQUEST_TEMPLATE/feature-pull-request.md

@@ -0,0 +1,45 @@
+---
+name: Feature
+about: Provide specifics about a PR for a feature
+title: 'PR/Feature: ____(provide a descriptive name)'
+labels: enhancement
+assignees: ''
+
+---
+<!-- IMPORTANT: Please do not create a PR without creating an issue first. -->
+
+<!-- Fields in **bold** are REQUIRED, fields in *italics* are OPTIONAL. -->
+
+**Description**
+<!-- A description of how this PR addresses the feature/enhancement. -->
+
+<!-- Add any linked issue(s) -->
+See #
+
+
+**ToDo List**
+
+<!-- Add any open questions and Pre-Merge TODOs. Use checkboxes. -->
+ - [ ] ToDo -->
+ - [ ] ToDo -->
+ - [ ] ToDo -->
+
+
+**Test plan**
+<!-- Explain how you tested this feature so that others can replicate it. -->
+<!-- Example: The exact commands you ran and their output, screenshots. -->
+
+
+**Update Changelog**
+<!-- Be brief, use imperative mood or simple noun phrases and add linked issues -->
+<!-- Examples: Improve verbosity of log messages #103 | GitHub actions for CI #105 -->
+
+- [ ] README.md, [changelog](../../README.md#changelog) <!-- update changelog here -->
+
+
+*Resources*
+<!-- Links to blog posts, StackOverflow, libraries or add-ons used to solve this problem. -->
+
+
+*Screenshots or additional context*
+<!-- Add any other context about the problem here and/or screenshots to help explain the problem. -->

.github/PULL_REQUEST_TEMPLATE/release-pull-request.md

@@ -0,0 +1,36 @@
+---
+name: Release
+about: Provide specifics about a PR for a release
+title: 'PR/Release: ____(provide a descriptive name)'
+assignees: ''
+
+---
+<!-- IMPORTANT: Please do not create a PR without creating an issue first. -->
+
+<!-- Fields in **bold** are REQUIRED, fields in *italics* are OPTIONAL. -->
+
+**Description**
+<!-- A description of the updates contained in this release. Example:  -->
+<!-- Incorporates Travis CI feature #91, summary for script_run #99, and root portal handling #88 -->
+
+Closes #
+
+
+**Update Changelog**
+<!-- List changes: be brief, use imperative mood or simple noun phrases and add linked issues -->
+<!-- Examples: Improve verbosity of log messages #103 | GitHub actions for CI #105 -->
+
+- [ ] README.md, [changelog](../../README.md#changelog) <!-- update changelog here -->
+
+
+**Bump version**
+
+v0.xx.x -> v0.xx.0
+
+- [ ] README.md, [installation instructions](../../README.md#installation-instructions)
+- [ ] [`setup.py`](../../setup.py)
+- [ ] [`requiam/__init__.py`](../../requiam/__init__.py)
+
+
+*Screenshots or additional context*
+<!-- Add any other context about this release. -->

.github/workflows/python-package.yml

@@ -25,7 +25,7 @@ jobs:
     if: "!contains(github.event.head_commit.message, 'ci skip') || !contains(github.event.head_commit.message, 'skip ci')"
     strategy:
       matrix:
-        python-version: ['3.7', '3.7.5', '3.8']
+        python-version: ['3.7', '3.8', '3.9']
 
     steps:
     - uses: actions/checkout@v2

README.md

@@ -64,7 +64,7 @@ These instructions will have the code running on your local or virtual machine.
 
 You will need the following to have a working copy of this software. See
 [installation](#installation-instructions) steps:
-1. Python (3.7.5)
+1. Python (>=3.7.9)
 2. [`pandas`](https://pandas.pydata.org/) ([0.25.3](https://pandas.pydata.org/pandas-docs/version/0.25.3/))
 3. [`ldap3`](https://ldap3.readthedocs.io/en/latest/) (2.6.1)
 4. [`requests`](https://requests.readthedocs.io/en/master/) (2.22.0)
@@ -74,14 +74,14 @@ You will need the following to have a working copy of this software. See
 
 #### Python and setting up a `conda` environment
 
-First, install a working version of Python (v3.7.5).  We recommend using the
+First, install a working version of Python (>=3.7.9).  We recommend using the
 [Anaconda](https://www.anaconda.com/distribution/) package installer.
 
 After you have Anaconda installed, you will want to create a separate `conda` environment
 and activate it:
 
 ```
-$ (sudo) conda create -n figshare_patrons python=3.7.5
+$ (sudo) conda create -n figshare_patrons python=3.7
 $ conda activate figshare_patrons
 ```
 
@@ -107,7 +107,7 @@ You can confirm installation via `conda list`
 (figshare_patrons) $ conda list requiam
 ```
 
-You should see that the version is `0.15.1`.
+You should see that the version is `0.16.0`.
 
 ### Configuration Settings
 
@@ -296,12 +296,19 @@ Currently, there are two GitHub Action workflows:
 A list of released features and their issue number(s).
 List is sorted from moderate to minor revisions for reach release.
 
-v0.15.0:
+v0.16.0:
+ * Merge `grouper_admin` and `grouper_query` modules #87
+ * Complete adoption of f-strings #118
+ * New pull request templates #120
+ * CI build testing for Python 3.9 #121
+
+v0.15.0 - v0.15.1:
  * GitHub actions for CI #105
  * Simplify scripts to use dictionary for configuration settings #104
  * Improve verbosity of log messages #103
  * Priority labels for project management #108
  * Add/update GitHub issue templates #110
+ * Bug: Fix f-string typo with batch load information #114
 
 v0.14.0:
  * Travis CI integration #91

requiam/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.15.1"
+__version__ = "0.16.0"
 
 
 class TimerClass(object):
@@ -44,4 +44,4 @@ def _stop(self):
         HH = int(sec // 3600)
         MM = int((sec // 60) - (HH * 60))
         SS = sec - (HH * 3600) - (MM * 60)
-        self.format = "Total time: {0: 02d} hours  {1: 02d} minutes  {2: .2f} seconds".format(HH, MM, SS)
+        self.format = f"Total time: {HH: 02d} hours  {MM: 02d} minutes  {SS: .2f} seconds"

requiam/commons.py

@@ -34,6 +34,39 @@ def figshare_stem(stem='', production=True):
     return stem_query
 
 
+def figshare_group(group, root_stem, production=True):
+    """
+    Purpose:
+      Construct Grouper figshare groups
+
+    :param group: str or int of group name. Cannot be empty
+    :param root_stem: str of associated stem folder for [group]
+    :param production: Bool to use production stem. Otherwise a stage/test is used. Default: True
+
+    :return grouper_group: str containing full Grouper path
+
+    Usage:
+      For active group, call as: figshare_group('active', '')
+        > 'arizona.edu:dept:LBRY:figshare:active'
+
+      For a quota group, call as: figshare_group('2147483648', 'quota')
+        > 'arizona.edu:dept:LBRY:figshare:quota:2147483648'
+      Note: group can be specified as an integer for quota cases
+
+      For a portal group, call as: figshare_group('sci_math', 'portal')
+        > 'arizona.edu:dept:LBRY:figshare:portal:sci_math'
+    """
+
+    if not group:
+        raise ValueError("WARNING: Empty [group]")
+
+    stem_query = figshare_stem(stem=root_stem, production=production)
+
+    grouper_group = f'{stem_query}:{group}'
+
+    return grouper_group
+
+
 def dict_load(config_file, vargs=None):
     """
     Purpose:

requiam/delta.py

@@ -22,8 +22,9 @@ class Delta(object):
 
     """
 
-    def __init__(self, ldap_members, grouper_query_instance, batch_size,
-                 batch_timeout, batch_delay, sync_max, log=None):
+    def __init__(self, ldap_members, grouper_query_dict,
+                 batch_size, batch_timeout, batch_delay, sync_max,
+                 log=None):
 
         if isinstance(log, type(None)):
             self.log = log_stdout()
@@ -33,7 +34,8 @@ def __init__(self, ldap_members, grouper_query_instance, batch_size,
         self.log.debug('entered')
 
         self.ldap_members = ldap_members
-        self.grouper_qry = grouper_query_instance
+        self.grouper_query_dict = grouper_query_dict
+        self.grouper_members = grouper_query_dict['members']
         self.batch_size = batch_size
         self.batch_timeout = batch_timeout
         self.batch_delay = batch_delay
@@ -47,19 +49,19 @@ def __init__(self, ldap_members, grouper_query_instance, batch_size,
         return
 
     def _common(self):
-        common = self.ldap_members & self.grouper_qry.members
+        common = self.ldap_members & self.grouper_members
 
         self.log.debug('finished common')
         return common
 
     def _adds(self):
-        adds = self.ldap_members - self.grouper_qry.members
+        adds = self.ldap_members - self.grouper_members
 
         self.log.debug('finished adds')
         return adds
 
     def _drops(self):
-        drops = self.grouper_qry.members - self.ldap_members
+        drops = self.grouper_members - self.ldap_members
 
         self.log.debug('finished drops')
         return drops
@@ -74,7 +76,8 @@ def synchronize(self):
             self.log.debug('finished synchronize')
             return
 
-        self.log.info(f"synchronizing ldap query results to {self.grouper_qry.grouper_group}")
+        self.log.info("synchronizing ldap query results to " +
+                      f"{self.grouper_query_dict['grouper_group']}")
         self.log.info(f"batch size = {self.batch_size}, " +
                       f"batch timeout = {self.batch_timeout} seconds, " +
                       f"batch delay = {self.batch_delay} seconds")
@@ -87,9 +90,9 @@ def synchronize(self):
             n_batches += 1
 
             start_t = datetime.datetime.now()
-            rsp = requests.post(self.grouper_qry.grouper_group_members_url,
-                                auth=(self.grouper_qry.grouper_user,
-                                      self.grouper_qry.grouper_password),
+            rsp = requests.post(self.grouper_query_dict['grouper_members_url'],
+                                auth=(self.grouper_query_dict['grouper_user'],
+                                      self.grouper_query_dict['grouper_password']),
                                 data=json.dumps({
                                     'WsRestDeleteMemberRequest': {
                                         'replaceAllExisting': 'F',
@@ -122,9 +125,9 @@ def synchronize(self):
             n_batches += 1
 
             start_t = datetime.datetime.now()
-            rsp = requests.put(self.grouper_qry.grouper_group_members_url,
-                               auth=(self.grouper_qry.grouper_user,
-                                     self.grouper_qry.grouper_password),
+            rsp = requests.put(self.grouper_query_dict['grouper_members_url'],
+                               auth=(self.grouper_query_dict['grouper_user'],
+                                     self.grouper_query_dict['grouper_password']),
                                data=json.dumps({
                                    'WsRestAddMemberRequest': {
                                        'replaceAllExisting': 'F',