New Validate tool error (v2.12.1): Fail during rule 3.10

[alwunder]

This error occurred on a partially complete database that I had previously been able to complete a validation on with v2.11.3+ of the tool (Issue #68). Database has not been edited since last successful check; DataSources, DMU, and Glossary tables are present but empty.

I have also tested the latest Validate tool on several complete databases and had no issues.

This version of the tool is up to date
Metadata not checked. File needs to be in XML format.
​
Looking at level 2 compliance
Rule 2.1 - Has required elements: nonspatial tables DataSources, 
        DescriptionOfMapUnits, GeoMaterialDict; feature dataset GeologicMap with 
        feature classes ContactsAndFaults and MapUnitPolys
Rule 2.2 - Required fields within required elements are present and correctly defined
2.3 All MapUnitPolys and ContactsAndFaults based feature classes obey Level 2 topology rules: 
        no internal gaps or overlaps in MapUnitPolys, boundaries of MapUnitPolys are covered by ContactsAndFaults
Topology check was skipped
2.4 All map units in MapUnitPolys have entries in DescriptionOfMapUnits table
2.5 No duplicate MapUnit values in DescriptionOfMapUnits table
2.6 Certain field values within required elements have entries in Glossary table
2.7 No duplicate Term values in Glossary table
2.8 All xxxSourceID values in required elements have entries in DataSources table
2.9 No duplicate DataSources_ID values in DataSources table
​
Looking at level 3 compliance
3.1 Table and field definitions conform to GeMS schema
3.2 All MapUnitPolys and ContactsAndFaults based feature classes obey Level 3 topology rules: 
        no ContactsAndFaults overlaps, self-overlaps, or self-intersections.
3.3 No missing required values
3.4 No missing terms in Glossary
3.5 No unnecessary terms in Glossary
3.6 No missing sources in DataSources
3.7 No unnecessary sources in DataSources
3.8 No map units without entries in DescriptionOfMapUnits
3.9 No unnecessary map units in DescriptionOfMapUnits
3.10 HierarchyKey values in DescriptionOfMapUnits are unique and well formed
Traceback (most recent call last):
  File "C:\GIS\TOOLS\GeMS\gems-tools-pro-2.11\Scripts\GeMS_ValidateDatabase.py", line 1609, in <module>
    ]
  File "C:\GIS\TOOLS\GeMS\gems-tools-pro-2.11\Scripts\GeMS_ValidateDatabase.py", line 1529, in main
    #     ver_table = db_dict["Version"]["catalogPath"]
ValueError: too many values to unpack (expected 2)

Failed script Validate Database...
Failed to execute (ValidateDatabase).

[ethoms-usgs]

Odd error again! The lines numbers don't help find the error (and one is a comment!) but I did find a typo related to checking 3.10 that the linter in VS Code did not find; doubly odd. Anyway, try the tool again from the Code button on the main page, not the latest release. I will publish a new one once we close this issue.

[alwunder]

I just tested it again and it errored in exactly the same way/place, it just incremented the line numbers up one but displayed the same lines of code... So I also added some junk comment lines to the code and it incremented the line numbers up but errored in the same two places. Odd indeed.

[ethoms-usgs]

Sounds like the cached vs in-situ problem. Before shutting everything down, could you try the right-click 'refresh' option on the toolbox?

[tfelger]

I am getting the same error but at slightly different line numbers, also on a gdb that validated using an older version of the tool.

Traceback (most recent call last):
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230906\Scripts\GeMS_ValidateDatabase.py", line 1610, in
]
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230906\Scripts\GeMS_ValidateDatabase.py", line 1530, in main
# ver_table = db_dict["Version"]["catalogPath"]
ValueError: too many values to unpack (expected 2)

This version of the tool is up to date
Metadata not checked. File needs to be in XML format.
​
Looking at level 2 compliance
Rule 2.1 - Has required elements: nonspatial tables DataSources,
DescriptionOfMapUnits, GeoMaterialDict; feature dataset GeologicMap with
feature classes ContactsAndFaults and MapUnitPolys
Rule 2.2 - Required fields within required elements are present and correctly defined
2.3 All MapUnitPolys and ContactsAndFaults based feature classes obey Level 2 topology rules:
no internal gaps or overlaps in MapUnitPolys, boundaries of MapUnitPolys are covered by ContactsAndFaults
No topology found in GeologicMap
Creating geodatabase for topology - C:\Data\TEMP\validate\Topology.gdb
Creating feature dataset GeologicMap
Copying feature classes
Creating topology GeologicMap_Topology
Adding topology rules to topology
Looking at validation results for errors
2.4 All map units in MapUnitPolys have entries in DescriptionOfMapUnits table
2.5 No duplicate MapUnit values in DescriptionOfMapUnits table
2.6 Certain field values within required elements have entries in Glossary table
2.7 No duplicate Term values in Glossary table
2.8 All xxxSourceID values in required elements have entries in DataSources table
2.9 No duplicate DataSources_ID values in DataSources table
​
Looking at level 3 compliance
3.1 Table and field definitions conform to GeMS schema
3.2 All MapUnitPolys and ContactsAndFaults based feature classes obey Level 3 topology rules:
no ContactsAndFaults overlaps, self-overlaps, or self-intersections.
3.3 No missing required values
3.4 No missing terms in Glossary
3.5 No unnecessary terms in Glossary
3.6 No missing sources in DataSources
3.7 No unnecessary sources in DataSources
3.8 No map units without entries in DescriptionOfMapUnits
3.9 No unnecessary map units in DescriptionOfMapUnits
3.10 HierarchyKey values in DescriptionOfMapUnits are unique and well formed
Traceback (most recent call last):
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230906\Scripts\GeMS_ValidateDatabase.py", line 1610, in
]
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230906\Scripts\GeMS_ValidateDatabase.py", line 1530, in main
# ver_table = db_dict["Version"]["catalogPath"]
ValueError: too many values to unpack (expected 2)
Failed script Validate Database...
Failed to execute (ValidateDatabase).

[alwunder]

Well, I've been through it several more times, tried on different unfinished databases, restarted the computer a couple of times, still no change. I have also run the tool on 8 other complete databases and they all validated no problem. So there definitely seems to be something up with the empty tables.

[ethoms-usgs]

Ok, now I see the error. I got a traceback to line 1775, which made sense. I have no idea why your error messages are showing those other lines.

I just pushed a fix to the main repo, again not the latest release. Try again.
Thanks for your patience!

[alwunder]

OK, success!! Thanks for tracking down the problem. It really stinks that the error reporting is not giving accurate info on the line numbers... makes it really difficult to debug!

I have tested the latest code on three different incomplete databases of varying complexity and all three finished validation and produced meaningful reports that I can use to complete the missing items. Good to go!

[ethoms-usgs]

Great! good to know.

Tracey?

[tfelger]

Thanks for checking, Evan!

I tested it on 2 gdbs - one that was created recently, and one that was created a couple years ago. The recently created one validates fine with the updated toolbox. The old one, which was level 3 compliant when published, now gives the error below.

Start Time: Thursday, September 7, 2023 9:43:05 AM
This version of the tool is up to date
Metadata not checked. File needs to be in XML format.
​
Looking at level 2 compliance
Rule 2.1 - Has required elements: nonspatial tables DataSources,
DescriptionOfMapUnits, GeoMaterialDict; feature dataset GeologicMap with
feature classes ContactsAndFaults and MapUnitPolys
Rule 2.2 - Required fields within required elements are present and correctly defined
2.3 All MapUnitPolys and ContactsAndFaults based feature classes obey Level 2 topology rules:
no internal gaps or overlaps in MapUnitPolys, boundaries of MapUnitPolys are covered by ContactsAndFaults
Topology in GeologicMap found
Validating topology
Looking at validation results for errors
Traceback (most recent call last):
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230907\Scripts\GeMS_ValidateDatabase.py", line 1610, in
]
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230907\Scripts\GeMS_ValidateDatabase.py", line 1357, in main
with arcpy.da.UpdateCursor(db_dict[table]["catalogPath"], field) as cursor:
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230907\Scripts\GeMS_ValidateDatabase.py", line 343, in check_topology
topology rules. No MapUnitPolys gaps or overlaps. No ContactsAndFaults overlaps, self-overlaps,
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230907\Scripts\topology.py", line 391, in eval_topology
[
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230907\Scripts\topology.py", line 392, in
f" Rule '{level_3_errors[r]}' is missing"
TypeError: list indices must be integers or slices, not str
Failed script Validate Database...
Failed to execute (ValidateDatabase).
Failed at Thursday, September 7, 2023 9:43:12 AM (Elapsed Time: 6.59 seconds)

[ethoms-usgs]

Ok, I found the possible cause. Try again downloading from the Code button.

[tfelger]

I downloaded the new version and tried it. Now I'm getting the following error:

Start Time: Friday, September 8, 2023 9:27:49 AM
This version of the tool is up to date
Metadata not checked. File needs to be in XML format.
​
Looking at level 2 compliance
Rule 2.1 - Has required elements: nonspatial tables DataSources,
DescriptionOfMapUnits, GeoMaterialDict; feature dataset GeologicMap with
feature classes ContactsAndFaults and MapUnitPolys
Rule 2.2 - Required fields within required elements are present and correctly defined
2.3 All MapUnitPolys and ContactsAndFaults based feature classes obey Level 2 topology rules:
no internal gaps or overlaps in MapUnitPolys, boundaries of MapUnitPolys are covered by ContactsAndFaults
Topology in GeologicMap found
Looking at validation results for errors
2.4 All map units in MapUnitPolys have entries in DescriptionOfMapUnits table
2.5 No duplicate MapUnit values in DescriptionOfMapUnits table
2.6 Certain field values within required elements have entries in Glossary table
2.7 No duplicate Term values in Glossary table
2.8 All xxxSourceID values in required elements have entries in DataSources table
2.9 No duplicate DataSources_ID values in DataSources table
​
Looking at level 3 compliance
3.1 Table and field definitions conform to GeMS schema
3.2 All MapUnitPolys and ContactsAndFaults based feature classes obey Level 3 topology rules:
no ContactsAndFaults overlaps, self-overlaps, or self-intersections.
3.3 No missing required values
3.4 No missing terms in Glossary
Traceback (most recent call last):
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230908\Scripts\GeMS_ValidateDatabase.py", line 1610, in
]
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230908\Scripts\GeMS_ValidateDatabase.py", line 1467, in main
open_report = guf.eval_bool(argv[10])
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230908\Scripts\GeMS_ValidateDatabase.py", line 514, in glossary_check
and not "guid" in f.name.lower()
File "C:\Data\ProjectData\GIS\DataModels\GeMS\ArcPro\gems-tools-pro-master_20230908\Scripts\GeMS_ValidateDatabase.py", line 160, in which_id
}
IndexError: list index out of range
Failed script Validate Database...
Failed to execute (ValidateDatabase).

[ethoms-usgs]

Wow, again, the line numbers mean nothing. What is going on!?
Can you send me the database? Or is it non-compliant in a way that I could easily modify one of my own for testing?