Unity Catalog Profiler Issue

[JacobMaldonado]

Hi, I am trying to run the profiler ingestion using Unity catalog.
I already ran the metadata one successfully. My issue is when I try to run it it fails with the following errors and even in debug mode I am unable to understand what is the issue.
Maybe I am missing some step.

I'm using version: 1.6.1

This is the log that I am getting in the Airflow Job.

25-01-20, 15:08:08 UTC] {local_task_job_runner.py:120} ▶ Pre task execution logs
[2025-01-20, 15:08:08 UTC] {server_mixin.py:74} INFO - OpenMetadata client running with Server version [1.6.1] and Client version [1.6.1.0]
[2025-01-20, 15:08:09 UTC] {ingestion_pipeline_mixin.py:53} DEBUG - Created Pipeline Status for pipeline dev unity catalog.62f91144-c1a8-42f5-8893-8cddfb54110a: runId='2ead5e6c-73f7-4cb3-85ed-b12ee19fa7f2' pipelineState=<PipelineState.running: 'running'> startDate=Timestamp(root=1737385688659) timestamp=Timestamp(root=1737385688659) endDate=None status=None
[2025-01-20, 15:08:09 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.service_spec.ServiceSpec
[2025-01-20, 15:08:10 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.metadata.UnitycatalogSource
[2025-01-20, 15:08:10 UTC] {metadata.py:76} INFO - Starting profiler for service dev unity catalog:unitycatalog
[2025-01-20, 15:08:10 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.sink.metadata_rest.MetadataRestSink
[2025-01-20, 15:08:11 UTC] {profiler.py:81} DEBUG - Sink type:metadata-rest, <class 'metadata.ingestion.sink.metadata_rest.MetadataRestSink'> configured
[2025-01-20, 15:08:11 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.connection.get_connection
[2025-01-20, 15:08:11 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.connection.test_connection
[2025-01-20, 15:08:11 UTC] {test_connections.py:203} INFO - Running CheckAccess...
[2025-01-20, 15:08:12 UTC] {test_connections.py:203} INFO - Running GetDatabases...
[2025-01-20, 15:08:12 UTC] {test_connections.py:203} INFO - Running GetSchemas...
[2025-01-20, 15:08:12 UTC] {test_connections.py:203} INFO - Running GetTables...
[2025-01-20, 15:08:13 UTC] {test_connections.py:203} INFO - Running GetViews...
[2025-01-20, 15:08:13 UTC] {test_connections.py:203} INFO - Running GetQueries...
[2025-01-20, 15:08:13 UTC] {test_connections.py:228} INFO - Test connection results:
[2025-01-20, 15:08:13 UTC] {test_connections.py:229} INFO - lastUpdatedAt=None status=<StatusType.Running: 'Running'> steps=[TestConnectionStepResult(name='CheckAccess', mandatory=True, passed=True, message=None, errorLog=None), TestConnectionStepResult(name='GetDatabases', mandatory=True, passed=True, message=None, errorLog=None), TestConnectionStepResult(name='GetSchemas', mandatory=True, passed=True, message=None, errorLog=None), TestConnectionStepResult(name='GetTables', mandatory=True, passed=True, message=None, errorLog=None), TestConnectionStepResult(name='GetViews', mandatory=False, passed=True, message=None, errorLog=None), TestConnectionStepResult(name='GetQueries', mandatory=False, passed=True, message=None, errorLog=None)]
[2025-01-20, 15:08:51 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.service_spec.ServiceSpec
[2025-01-20, 15:08:51 UTC] {importer.py:129} DEBUG - Importing: metadata.profiler.interface.sqlalchemy.unity_catalog.profiler_interface.UnityCatalogProfilerInterface
[2025-01-20, 15:08:51 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.service_spec.ServiceSpec
[2025-01-20, 15:08:51 UTC] {importer.py:129} DEBUG - Importing: metadata.sampler.sqlalchemy.sampler.SQASampler
[2025-01-20, 15:08:51 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.connection.get_connection
[2025-01-20, 15:08:51 UTC] {importer.py:129} DEBUG - Importing: metadata.ingestion.source.database.unitycatalog.connection.get_connection
[2025-01-20, 15:08:56 UTC] {core.py:483} DEBUG - Computing profile metrics for dev unity catalog.lakehouse_dev.gold.dim_colada...
[2025-01-20, 15:08:56 UTC] {status.py:91} WARNING - Unexpected exception processing entity dev unity catalog.lakehouse_dev.gold.dim_colada: 2 validation errors for ThreadPoolMetrics
table.function-after[parse_name(), Table]
  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/model_type
table.is-instance[DeclarativeMeta]
  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/is_instance_of
[2025-01-20, 15:08:56 UTC] {status.py:92} DEBUG - Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/processor.py", line 63, in _run
    profile: ProfilerResponse = profiler_runner.process()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 486, in process
    self.compute_metrics()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 468, in compute_metrics
    self.profile_entity()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 450, in profile_entity
    table_metrics = self._prepare_table_metrics()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 331, in _prepare_table_metrics
    ThreadPoolMetrics(
  File "/home/airflow/.local/lib/python3.10/site-packages/pydantic/main.py", line 176, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for ThreadPoolMetrics
table.function-after[parse_name(), Table]
  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/model_type
table.is-instance[DeclarativeMeta]
  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/is_instance_of
[2025-01-20, 15:08:56 UTC] {ingestion_pipeline_mixin.py:53} DEBUG - Created Pipeline Status for pipeline dev unity catalog.62f91144-c1a8-42f5-8893-8cddfb54110a: runId='2ead5e6c-73f7-4cb3-85ed-b12ee19fa7f2' pipelineState=<PipelineState.success: 'success'> startDate=Timestamp(root=1737385688659) timestamp=Timestamp(root=1737385688659) endDate=Timestamp(root=1737385736814) status=IngestionStatus(root=[StepSummary(name='OpenMetadata Service', records=1, updated_records=0, warnings=0, errors=0, filtered=703, failures=None), StepSummary(name='Profiler', records=0, updated_records=0, warnings=0, errors=1, filtered=0, failures=[StackTraceError(name='dev unity catalog.lakehouse_dev.gold.dim_colada', error='Unexpected exception processing entity dev unity catalog.lakehouse_dev.gold.dim_colada: 2 validation errors for ThreadPoolMetrics\ntable.function-after[parse_name(), Table]\n  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/model_type\ntable.is-instance[DeclarativeMeta]\n](https://errors.pydantic.dev/2.7/v/model_type/ntable.is-instance[DeclarativeMeta]/n)  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]\n    For further information visit https://errors.pydantic.dev/2.7/v/is_instance_of', stackTrace='Traceback (most recent call last):\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/processor.py", line 63, in _run\n    profile: ProfilerResponse = profiler_runner.process()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 486, in process\n    self.compute_metrics()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 468, in compute_metrics\n    self.profile_entity()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 450, in profile_entity\n    table_metrics = self._prepare_table_metrics()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 331, in _prepare_table_metrics\n    ThreadPoolMetrics(\n  File "/home/airflow/.local/lib/python3.10/site-packages/pydantic/main.py", line 176, in __init__\n    self.__pydantic_validator__.validate_python(data, self_instance=self)\npydantic_core._pydantic_core.ValidationError: 2 validation errors for ThreadPoolMetrics\ntable.function-after[parse_name(), Table]\n  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/model_type\ntable.is-instance[DeclarativeMeta]\n](https://errors.pydantic.dev/2.7/v/model_type/ntable.is-instance[DeclarativeMeta]/n)  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/is_instance_of\n')]),](https://errors.pydantic.dev/2.7/v/is_instance_of/n')]),) StepSummary(name='OpenMetadata', records=0, updated_records=0, warnings=0, errors=0, filtered=0, failures=None)])
[2025-01-20, 15:08:57 UTC] {ingestion_pipeline_mixin.py:53} DEBUG - Created Pipeline Status for pipeline dev unity catalog.62f91144-c1a8-42f5-8893-8cddfb54110a: runId='2ead5e6c-73f7-4cb3-85ed-b12ee19fa7f2' pipelineState=<PipelineState.failed: 'failed'> startDate=Timestamp(root=1737385688659) timestamp=Timestamp(root=1737385688659) endDate=Timestamp(root=1737385737030) status=IngestionStatus(root=[StepSummary(name='OpenMetadata Service', records=1, updated_records=0, warnings=0, errors=0, filtered=703, failures=[]), StepSummary(name='Profiler', records=0, updated_records=0, warnings=0, errors=1, filtered=0, failures=[StackTraceError(name='dev unity catalog.lakehouse_dev.gold.dim_colada', error='Unexpected exception processing entity dev unity catalog.lakehouse_dev.gold.dim_colada: 2 validation errors for ThreadPoolMetrics\ntable.function-after[parse_name(), Table]\n  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/model_type\ntable.is-instance[DeclarativeMeta]\n](https://errors.pydantic.dev/2.7/v/model_type/ntable.is-instance[DeclarativeMeta]/n)  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]\n    For further information visit https://errors.pydantic.dev/2.7/v/is_instance_of', stackTrace='Traceback (most recent call last):\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/processor.py", line 63, in _run\n    profile: ProfilerResponse = profiler_runner.process()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 486, in process\n    self.compute_metrics()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 468, in compute_metrics\n    self.profile_entity()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 450, in profile_entity\n    table_metrics = self._prepare_table_metrics()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 331, in _prepare_table_metrics\n    ThreadPoolMetrics(\n  File "/home/airflow/.local/lib/python3.10/site-packages/pydantic/main.py", line 176, in __init__\n    self.__pydantic_validator__.validate_python(data, self_instance=self)\npydantic_core._pydantic_core.ValidationError: 2 validation errors for ThreadPoolMetrics\ntable.function-after[parse_name(), Table]\n  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/model_type\ntable.is-instance[DeclarativeMeta]\n](https://errors.pydantic.dev/2.7/v/model_type/ntable.is-instance[DeclarativeMeta]/n)  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/is_instance_of\n')]),](https://errors.pydantic.dev/2.7/v/is_instance_of/n')]),) StepSummary(name='OpenMetadata', records=0, updated_records=0, warnings=0, errors=0, filtered=0, failures=[])])
[2025-01-20, 15:08:57 UTC] {taskinstance.py:2905} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 400, in wrapper
    return func(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 235, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 252, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/openmetadata_managed_apis/workflows/ingestion/profiler.py", line 49, in profiler_workflow
    workflow.raise_from_status()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/workflow_status_mixin.py", line 132, in raise_from_status
    raise err
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/workflow_status_mixin.py", line 129, in raise_from_status
    self.raise_from_status_internal(raise_warnings)
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/base.py", line 197, in raise_from_status_internal
    raise WorkflowExecutionError(
metadata.config.common.WorkflowExecutionError: Profiler reported errors: Profiler Summary: [0 Records, [0 Updated Records, 0 Warnings, 1 Errors, 0 Filtered]
[2025-01-20, 15:08:57 UTC] {common.py:306} INFO - Sending failed status from callback...
[2025-01-20, 15:08:57 UTC] {server_mixin.py:74} INFO - OpenMetadata client running with Server version [1.6.1] and Client version [1.6.1.0]
[2025-01-20, 15:08:57 UTC] {common.py:312} INFO - Sending status to Ingestion Pipeline dev unity catalog.62f91144-c1a8-42f5-8893-8cddfb54110a
[2025-01-20, 15:08:57 UTC] {ingestion_pipeline_mixin.py:53} DEBUG - Created Pipeline Status for pipeline dev unity catalog.62f91144-c1a8-42f5-8893-8cddfb54110a: runId='2ead5e6c-73f7-4cb3-85ed-b12ee19fa7f2' pipelineState=<PipelineState.failed: 'failed'> startDate=Timestamp(root=1737385688659) timestamp=Timestamp(root=1737385688659) endDate=Timestamp(root=1737385737305) status=IngestionStatus(root=[StepSummary(name='OpenMetadata Service', records=1, updated_records=0, warnings=0, errors=0, filtered=703, failures=[]), StepSummary(name='Profiler', records=0, updated_records=0, warnings=0, errors=1, filtered=0, failures=[StackTraceError(name='dev unity catalog.lakehouse_dev.gold.dim_colada', error='Unexpected exception processing entity dev unity catalog.lakehouse_dev.gold.dim_colada: 2 validation errors for ThreadPoolMetrics\ntable.function-after[parse_name(), Table]\n  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/model_type\ntable.is-instance[DeclarativeMeta]\n](https://errors.pydantic.dev/2.7/v/model_type/ntable.is-instance[DeclarativeMeta]/n)  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]\n    For further information visit https://errors.pydantic.dev/2.7/v/is_instance_of', stackTrace='Traceback (most recent call last):\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/processor.py", line 63, in _run\n    profile: ProfilerResponse = profiler_runner.process()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 486, in process\n    self.compute_metrics()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 468, in compute_metrics\n    self.profile_entity()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 450, in profile_entity\n    table_metrics = self._prepare_table_metrics()\n  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/profiler/processor/core.py", line 331, in _prepare_table_metrics\n    ThreadPoolMetrics(\n  File "/home/airflow/.local/lib/python3.10/site-packages/pydantic/main.py", line 176, in __init__\n    self.__pydantic_validator__.validate_python(data, self_instance=self)\npydantic_core._pydantic_core.ValidationError: 2 validation errors for ThreadPoolMetrics\ntable.function-after[parse_name(), Table]\n  Input should be a valid dictionary or instance of Table [type=model_type, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/model_type\ntable.is-instance[DeclarativeMeta]\n](https://errors.pydantic.dev/2.7/v/model_type/ntable.is-instance[DeclarativeMeta]/n)  Input should be an instance of DeclarativeMeta [type=is_instance_of, input_value=None, input_type=NoneType]\n    For further information visit [https://errors.pydantic.dev/2.7/v/is_instance_of\n')]),](https://errors.pydantic.dev/2.7/v/is_instance_of/n')]),) StepSummary(name='OpenMetadata', records=0, updated_records=0, warnings=0, errors=0, filtered=0, failures=[])])
[2025-01-20, 15:08:57 UTC] {standard_task_runner.py:110} ERROR - Failed to execute job 61 for task profiler_task (Profiler reported errors: Profiler Summary: [0 Records, [0 Updated Records, 0 Warnings, 1 Errors, 0 Filtered]; 64910)
[2025-01-20, 15:08:57 UTC] {local_task_job_runner.py:240} INFO - Task exited with return code 1

[Prajwal214]

@JacobMaldonado This is fixed in v1.6.2 release

[JacobMaldonado]

Thank you so much.