Skip to content

list_wings returns empty when collection has >100K records (SQLite variable limit) #688

@lukeleekr

Description

@lukeleekr

Bug

list_wings (and likely list_rooms, get_taxonomy, status) returns {} even when the palace has 114,971 drawers with valid wing/room metadata.

Root Cause

The MCP server calls col.get(include=["metadatas"]) on the entire collection at once. With 114K+ records, this exceeds SQLite's SQL variable limit in ChromaDB 1.x. The except: pass block silently swallows the error, returning empty results.

Environment

  • mempalace 3.0.0
  • chromadb 1.5.6
  • macOS (Apple Silicon)
  • Python 3.12

Expected Behavior

list_wings should return wing names with drawer counts regardless of collection size.

Suggested Fix

Instead of loading all records, either:

  1. Paginate col.get() with offset/limit
  2. Query SQLite directly for distinct wing values + counts (most efficient)
  3. Use ChromaDB's where filter to query one wing at a time using the configured topic_wings from config.json

Also: replace except: pass with proper error logging so failures aren't silently swallowed.

Separate Issue (FYI)

After upgrading from pre-1.x to ChromaDB 1.5.6, a stale index_metadata.pickle file in the HNSW segment directory caused search to fail with Error loading hnsw index. Deleting this legacy file (from the old Python hnswlib format) resolved it. The Rust HNSW backend in 1.5.6 doesn't expect this file. Consider adding a migration step or startup check that removes legacy pickle files.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions