Skip to content

feat(gooddata-sdk): [AUTO] Add AI Lake database instances, pipe tables, and StarRocks service management#1567

Open
yenkins-admin wants to merge 1 commit intomasterfrom
auto/openapi-sync-C003-20260420-r82171
Open

feat(gooddata-sdk): [AUTO] Add AI Lake database instances, pipe tables, and StarRocks service management#1567
yenkins-admin wants to merge 1 commit intomasterfrom
auto/openapi-sync-C003-20260420-r82171

Conversation

@yenkins-admin
Copy link
Copy Markdown
Contributor

Summary

Added AI Lake database instance, pipe table, and StarRocks service management to the Python SDK. Created a new catalog/ai_lake/ domain with entity models (CatalogDatabaseInstance, CatalogPipeTable, CatalogPipeTableSummary, CatalogServiceInfo) and request models (CatalogProvisionDatabaseInstanceRequest, CatalogCreatePipeTableRequest), plus a CatalogAiLakeService. Added CatalogDataSourceAiLakehouse for the new AILAKEHOUSE data source type. Wired AILakeApi and AILakePipeTablesApi into GoodDataApiClient and registered CatalogAiLakeService as GoodDataSdk.catalog_ai_lake. All new classes exported from init.py. Unit tests cover all new models and serialization.

Impact: new_feature | Services: gooddata-metadata-client

Files changed

  • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/__init__.py
  • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/entity_model/__init__.py
  • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/entity_model/database_instance.py
  • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/entity_model/pipe_table.py
  • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/entity_model/service_info.py
  • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/service.py
  • packages/gooddata-sdk/src/gooddata_sdk/catalog/data_source/entity_model/data_source.py
  • packages/gooddata-sdk/src/gooddata_sdk/client.py
  • packages/gooddata-sdk/src/gooddata_sdk/sdk.py
  • packages/gooddata-sdk/src/gooddata_sdk/__init__.py
  • packages/gooddata-sdk/tests/catalog/test_catalog_ai_lake.py

Agent decisions

Decisions (5)

New catalog domain vs extending data_source — Created separate catalog/ai_lake/ domain

  • Alternatives: Add AI Lake methods to CatalogDataSourceService, Add new files inside catalog/data_source/
  • Why: AI Lake is a distinct service domain (database provisioning, pipe tables, StarRocks services) unrelated to data source CRUD. A dedicated domain follows the same pattern as other top-level services.

Combined AILakeApi vs separate APIs — Use AILakeApi for database/service operations and AILakePipeTablesApi for pipe tables

  • Alternatives: Use only individual AILakeDatabasesApi + AILakePipeTablesApi
  • Why: AILakeApi is the only AI Lake API exported from apis/init.py; AILakePipeTablesApi imported directly since it is missing from the apis module.

Custom from_api() vs Base.from_api() — Custom from_api() on each entity class using dict access

  • Alternatives: Inherit from Base and use cattrs.structure
  • Why: AI Lake models are plain ModelNormal objects, not JsonApi entities. Custom from_api() is explicit and avoids cattrs registration complexity.

CatalogDataSourceAiLakehouse credentials — Use _NoCredentials factory (same as CatalogDataSourceGdStorage)

  • Alternatives: Require explicit credentials, Add a new credential type
  • Why: AILAKEHOUSE is a GoodData-managed data source like GDSTORAGE; _NoCredentials is the established pattern.

PipeTable complex sub-models — Store columns and nested configs as list[dict[str, Any]]

  • Alternatives: Model PipeTableDistributionConfig, PipeTableKeyConfig etc. as attrs classes
  • Why: Complex oneOf sub-models not required for initial implementation; dict allows incremental modelling.
Assumptions to verify (5)
  • AILAKEHOUSE type already present in JsonApiDataSourceInAttributes.allowed_values — confirmed by grep.
  • AILakePipeTablesApi is NOT exported from gooddata_api_client.apis but importable from the direct module path.
  • to_dict(camel_case=False) on API client model objects returns snake_case keys matching from_api() field access.
  • DatabaseInstance.status field from the diff enum is in the schema description but not yet in the generated model openapi_types.
  • PipeTable list endpoint returns PipeTableSummary objects — confirmed by ListPipeTablesResponse.openapi_types.
Risks (3)
  • to_dict(camel_case=False) key mapping may miss 'storageIds' if API client serializes differently
  • TestCatalogDataSourceAiLakehouse will fail if AILAKEHOUSE is absent from JsonApiDataSourceInAttributes.allowed_values at test time
  • CatalogPipeTable.from_api may raise KeyError if server omits required fields like polling_interval_seconds
Layers touched (3)
  • entity_model — New AI Lake entity/request models; CatalogDataSourceAiLakehouse added
    • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/entity_model/database_instance.py
    • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/entity_model/pipe_table.py
    • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/entity_model/service_info.py
    • packages/gooddata-sdk/src/gooddata_sdk/catalog/data_source/entity_model/data_source.py
  • public_api — CatalogAiLakeService; AILakeApi + AILakePipeTablesApi in client; GoodDataSdk.catalog_ai_lake property; all exports
    • packages/gooddata-sdk/src/gooddata_sdk/catalog/ai_lake/service.py
    • packages/gooddata-sdk/src/gooddata_sdk/client.py
    • packages/gooddata-sdk/src/gooddata_sdk/sdk.py
    • packages/gooddata-sdk/src/gooddata_sdk/__init__.py
  • tests — Unit tests for entity model construction and request serialization
    • packages/gooddata-sdk/tests/catalog/test_catalog_ai_lake.py

Source commits (gdc-nas)

  • ea9f6a2 Merge pull request #21474 from gooddata/dho/cq-104-pipe
  • 81eb97a Merge pull request #21382 from gooddata/dho/cq-104-arrow-api-2
  • 78be9c1 Merge pull request #21285 from gooddata/dho/cq-104-arrow-api
  • afe2a10 Merge pull request #21000 from gooddata/dho/cq-1970-iceberg-native
  • 88d3e56 Merge pull request #21023 from gooddata/dho/cq-1970-services
  • ace834d Merge pull request #21012 from gooddata/dho/cq-1970-catalog
  • 56188cd Merge pull request #21019 from gooddata/dho/cq-1970-namespace
  • 9a28c06 Merge pull request #21026 from gooddata/dho/cq-1970-task-cleanup
  • f2dc57c Merge pull request #20972 from gooddata/dho/cq-1970-model
  • 6e5e782 Merge pull request #20812 from gooddata/dho/cq-1887-database-instance-list
  • e8261d6 Merge pull request #20822 from gooddata/dho/cq-1887-paging
  • ad00dc1 Merge pull request #20896 from gooddata/dho/cq-1887-consistent
  • d91dfaf Merge pull request #20766 from gooddata/dho/cq-1967-provider
  • e36bf71 Merge pull request #20778 from gooddata/dho/cq-1967-provider-aws
  • 8cb3047 Merge pull request #20785 from gooddata/dho/cq-1967-spi
  • 1056335 Merge pull request #20679 from gooddata/dho/cq-1967-service-rename
  • 0568730 Merge pull request #20869 from gooddata/dho/cq-2050-ailake
  • b4e5b11 Merge pull request #20276 from gooddata/dho/cq-1987-ai-lakehouse-ds
OpenAPI diff
+      "CreatePipeTableRequest": {
+        "properties": {
+          "sourceDataSourceId": { "type": "string" },
+          "sourceSchemaName": { "type": "string" },
+          "sourceTableName": { "type": "string" }
+        }
+      },
+      "DatabaseInstance": {
+        "properties": {
+          "id": { "type": "string" },
+          "name": { "type": "string" },
+          "status": { "enum": ["CREATING","RUNNING","STOPPING","STOPPED","DELETING","FAILED"] }
+        }
+      },
+      "PipeTable": { ... },
+      "ServiceInfo": { ... },
+    paths:
+      "/api/v1/entities/dataSources": datasource type enum additions:
+        +"AILAKEHOUSE"

Workflow run


Generated by SDK OpenAPI Sync workflow

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 20, 2026

Codecov Report

❌ Patch coverage is 86.56716% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.73%. Comparing base (c00ca3b) to head (08ebaed).
⚠️ Report is 170 commits behind head on master.

Files with missing lines Patch % Lines
...ta-sdk/src/gooddata_sdk/catalog/ai_lake/service.py 52.77% 17 Missing ⚠️
packages/gooddata-sdk/src/gooddata_sdk/sdk.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #1567       +/-   ##
===========================================
+ Coverage    0.00%   78.73%   +78.73%     
===========================================
  Files         158      234       +76     
  Lines       11048    15539     +4491     
===========================================
+ Hits            0    12234    +12234     
+ Misses      11048     3305     -7743     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant