Real-Time Normalized Cache with TanStack Query

The useNormalizedQuery hook provides instant, event-driven cache updates with server-side filtering. Built with 2025 best practices including runtime validation (Valibot), type-safe merging (ts-deepmerge), and proper subscription cleanup.

Overview

useNormalizedQuery wraps TanStack Query to add real-time capabilities:

Instant updates across all devices via WebSocket events
Server-side filtering reduces network traffic by 90%+
Client-side safety ensures correctness even with unrelated events
Proper cleanup prevents connection leaks
Runtime validation catches malformed events
Type-safe merging preserves data integrity

Architecture

Basic Usage

Directory Listing

import { useNormalizedQuery } from "@sd/ts-client";

function DirectoryView({ path }: { path: SdPath }) {
	const { data, isLoading } = useNormalizedQuery({
		query: "files.directory_listing",
		input: { path },
		resourceType: "file",
		pathScope: path,
		includeDescendants: false, // Only direct children
	});

	if (isLoading) return <Spinner />;

	return (
		<div>
			{data?.files?.map((file) => <FileCard key={file.id} file={file} />)}
		</div>
	);
}

What happens:

Initial query fetches directory listing
Hook subscribes to file events for this path (exact mode)
When files are created/updated, events arrive instantly
Cache updates atomically
UI re-renders with new data

Media View (Recursive)

function MediaGallery({ path }: { path: SdPath }) {
	const { data } = useNormalizedQuery({
		query: "files.media_listing",
		input: { path, include_descendants: true },
		resourceType: "file",
		pathScope: path,
		includeDescendants: true, // All media in subtree
	});

	return (
		<Grid>
			{data?.files?.map((file) => <MediaThumbnail key={file.id} file={file} />)}
		</Grid>
	);
}

Global Resources

function LocationsList() {
	const { data } = useNormalizedQuery({
		query: "locations.list",
		input: null,
		resourceType: "location",
		// No pathScope - locations are global resources
	});

	return (
		<ul>{data?.locations?.map((loc) => <li key={loc.id}>{loc.name}</li>)}</ul>
	);
}

Single Resource Queries

function FileInspector({ fileId }: { fileId: string }) {
	const { data: file } = useNormalizedQuery({
		query: "files.by_id",
		input: { file_id: fileId },
		resourceType: "file",
		resourceId: fileId, // Only events for this file
	});

	return (
		<div>
			<h1>{file?.name}</h1>
			{/* Updates instantly when thumbnails generate */}
			{file?.sidecars?.map((sidecar) => (
				<Thumbnail key={sidecar.id} src={sidecar.url} />
			))}
		</div>
	);
}

API Reference

Options

interface UseNormalizedQueryOptions<I> {
	// Query method to call (e.g., "files.directory_listing")
	query: string;

	// Input for the query
	input: I;

	// Resource type for event filtering (e.g., "file", "location")
	resourceType: string;

	// Whether query is enabled (default: true)
	enabled?: boolean;

	// Optional path scope for server-side filtering
	pathScope?: SdPath;

	// Whether to include descendants (recursive) or only direct children (exact)
	// Default: false (exact matching)
	includeDescendants?: boolean;

	// Resource ID for single-resource queries
	resourceId?: string;
}

Path Filtering Modes

Exact Mode (Default)

Only events for files directly in the specified directory:

pathScope: { Physical: { device_slug: 'my-mac', path: '/Photos' } },
includeDescendants: false // or omit (default)

Behavior:

File in /Photos/image.jpg → ✓ Included
File in /Photos/Vacation/beach.jpg → ✗ Excluded
Directory /Photos/Vacation → ✗ Excluded

Recursive Mode

All events for files anywhere under the specified directory:

pathScope: { Physical: { device_slug: 'my-mac', path: '/Photos' } },
includeDescendants: true

Behavior:

File in /Photos/image.jpg → ✓ Included
File in /Photos/Vacation/beach.jpg → ✓ Included
File in /Photos/Vacation/Cruise/pic.jpg → ✓ Included

Server-Side Filtering

How It Works

Each hook creates a filtered subscription on the backend:

client.subscribeFiltered({
	resource_type: "file", // Only file events
	path_scope: "/Desktop", // Only this path
	include_descendants: false, // Exact mode
	library_id: "abc-123", // Current library
});

Backend applies filters before sending events:

✓ resource_type matches?
✓ library_id matches?
✓ path_scope matches? (with include_descendants mode)
✓ resourceId matches? (if specified)

Result: Only matching events are transmitted over the network.

Filter Logic

Exact Mode:

Event has affected_paths: [
  "/Desktop/file.txt",           // File path
  "/Desktop"                     // Parent directory
]

Subscription path_scope: "/Desktop"
include_descendants: false

Check: Does affected_paths contain "/Desktop" exactly?
Result: YES → Forward event

Recursive Mode:

Event has affected_paths: [
  "/Desktop/Subfolder/file.txt",
  "/Desktop/Subfolder"
]

Subscription path_scope: "/Desktop"
include_descendants: true

Check: Does "/Desktop/Subfolder" start with "/Desktop"?
Result: YES → Forward event

Client-Side Safety Filtering

Even with server-side filtering, the client applies a safety filter to batch events:

// Server forwards batch if ANY file matches
// Client filters to ONLY files that match

Batch has 100 files:
- 10 in /Desktop/ (direct children)
- 90 in /Desktop/Subfolder/ (subdirectories)

Server: Has 1 direct child → forward entire batch
Client: Filter batch → keep only 10 direct children
Cache: Contains only 10 files ✓

This ensures correctness even if server-side filtering has edge cases.

Event Types

ResourceChanged (Single)

{
  ResourceChanged: {
    resource_type: "location",
    resource: {
      id: "uuid",
      name: "Photos",
      path: "/Users/me/Photos",
      // ... full resource data
    },
    metadata: {
      no_merge_fields: ["sd_path"],
      affected_paths: [],
      alternate_ids: []
    }
  }
}

ResourceChangedBatch (Multiple)

{
  ResourceChangedBatch: {
    resource_type: "file",
    resources: [
      { id: "1", name: "photo1.jpg", ... },
      { id: "2", name: "photo2.jpg", ... }
    ],
    metadata: {
      no_merge_fields: ["sd_path"],
      affected_paths: [
        { Physical: { device_slug: "mac", path: "/Desktop/photo1.jpg" } },
        { Physical: { device_slug: "mac", path: "/Desktop" } },
        { Content: { content_id: "uuid" } }
      ],
      alternate_ids: []
    }
  }
}

ResourceDeleted

{
  ResourceDeleted: {
    resource_type: "location",
    resource_id: "uuid"
  }
}

Refresh (Invalidate All)

"Refresh";

Triggers queryClient.invalidateQueries() to refetch all data.

Deep Merge Behavior

Uses ts-deepmerge for type-safe, configurable merging:

// Existing cache
{
  id: "1",
  name: "Photos",
  metadata: { size: 1024, created_at: "2024-01-01" }
}

// Incoming event (partial update)
{
  id: "1",
  name: "My Photos",
  metadata: { size: 2048 }
}

// Result after merge
{
  id: "1",
  name: "My Photos",      // Updated
  metadata: {
    size: 2048,           // Updated
    created_at: "2024-01-01"  // Preserved ✓
  }
}

No-Merge Fields

Some fields should be replaced entirely, not merged:

metadata: {
	no_merge_fields: ["sd_path"];
}

// sd_path is replaced entirely, not deep merged
// This prevents incorrect path combinations

Runtime Validation

All events are validated with Valibot before processing:

const ResourceChangedSchema = v.object({
  ResourceChanged: v.object({
    resource_type: v.string(),
    resource: v.any(),
    metadata: v.nullish(v.object({ ... }))
  })
});

// Invalid events are logged and ignored
// Prevents crashes from malformed backend data

Subscription Multiplexing

Multiple hooks with identical filters automatically share a single backend subscription:

// Component A
function LocationsList() {
  useNormalizedQuery({
    query: 'locations.list',
    resourceType: 'location',
  });
}

// Component B (mounted at same time)
function LocationsDropdown() {
  useNormalizedQuery({
    query: 'locations.list',
    resourceType: 'location',
  });
}

// Result: Only 1 backend subscription created!
// Both hooks receive events from the same connection.

How it works:

First hook creates subscription with filter {resource_type: "location", library_id: "abc"}
Subscription manager generates key from filter: {"resource_type":"location","library_id":"abc"}
Second hook with same filter reuses existing subscription
Events broadcast to all listeners
When both unmount, subscription cleaned up automatically

Benefits:

Eliminates duplicate subscriptions during render cycles
Reduces backend load (fewer Unix socket connections)
Faster subscription setup (reuses existing connection)
Automatic reference counting prevents premature cleanup

Subscription Cleanup

Subscriptions are properly cleaned up when components unmount:

useEffect(() => {
	let unsubscribe: (() => void) | undefined;

	client.subscribeFiltered(filter, handleEvent).then((unsub) => {
		unsubscribe = unsub;
	});

	return () => {
		unsubscribe?.(); // Closes WebSocket subscription
	};
}, [dependencies]);

Cleanup process:

React calls cleanup function
Frontend stops listening to events
Tauri sends Unsubscribe request to daemon
Daemon closes subscription
Unix socket connection closed

Result: No connection leaks, no memory leaks.

Performance

Event Reduction

Indexing 10,000 files:

Without filtering:
- Each hook receives: 10,000 events
- Total transmitted: 50,000 events (5 hooks × 10,000)
- Result: UI lag, slow

With filtering:
- Desktop hook: 100 events (1%)
- Movies hook: 500 events (5%)
- Inspector: 1-5 events (0.05%)
- Total transmitted: ~600 events
- Result: Zero lag

Connection Management

Multiplexing: Multiple hooks with identical filters share one backend subscription
Reference counting: Subscriptions cleaned up when last hook unmounts
Deduplication: Eliminates duplicate subscriptions during render cycles
Monitoring: Check client.getSubscriptionStats() for active subscriptions

Testing

Test Coverage

Rust (Backend):

9/9 event filtering tests passing
Validates exact vs recursive modes
Tests all path types (Physical, Content, Cloud, Sidecar)

TypeScript (Frontend):

5/5 integration tests passing
Uses real backend event fixtures
Validates filtering and cache updates
Proves correctness with actual production code

Run Tests

# Rust tests
cargo test --test event_filtering_test

# TypeScript tests
cd packages/ts-client && bun test

# Generate new fixtures from backend
cargo test --test normalized_cache_fixtures_test

Best Practices

Always Scope File Queries

// Good
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path, // Server filters efficiently
});

// Bad - will skip subscription
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	// Missing pathScope! Subscription skipped to prevent overload
});

Use Correct Mode for View Type

// Directory view - exact mode
includeDescendants: false; // Only direct children

// Media gallery - recursive mode
includeDescendants: true; // All media in subtree

// Search results - recursive mode
includeDescendants: true; // All matching files

Combine with TanStack Query Options

const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path,
	// TanStack Query options
	enabled: !!path,
	staleTime: 5 * 60 * 1000,
	refetchOnWindowFocus: true,
});

Advanced Usage

Content-Addressed Files

Files use Content-based sd_path but have Physical paths in alternate_paths:

// File structure
{
  sd_path: { Content: { content_id: "uuid" } },
  alternate_paths: [
    { Physical: { device_slug: "mac", path: "/Desktop/file.txt" } }
  ]
}

// Client-side filtering uses alternate_paths for path matching
// This enables deduplication while maintaining path filtering

Multiple Instances

Multiple files with same content have different IDs:

// file1.txt (original)
{ id: "1", content_identity: { uuid: "abc" } }

// file2.txt (duplicate)
{ id: "2", content_identity: { uuid: "abc" } }

// Both update when content is processed

Debugging

Enable Logging

// Check console for:
// "[useNormalizedQuery] Invalid event: ..." - Validation failures
// "[TauriTransport] Unsubscribing: ..." - Cleanup events

Monitor Subscriptions

# Backend logs show subscription lifecycle
RUST_LOG=sd_core::infra::daemon::rpc=debug cargo run -p spacedrive-tauri

# Look for:
# "New subscription created: ..." - Subscription started
# "Subscription cancelled: ..." - Cleanup triggered
# "Unsubscribe sent successfully" - Connection closed

Frontend subscription stats:

import { useSpacedriveClient } from '@sd/ts-client';

function DebugPanel() {
  const client = useSpacedriveClient();
  const stats = client.getSubscriptionStats();

  console.log(`Active subscriptions: ${stats.activeSubscriptions}`);
  stats.subscriptions.forEach(sub => {
    console.log(`  ${sub.key}: ${sub.refCount} hooks, ${sub.listenerCount} listeners`);
  });
}

Inspect Cache

import { useQueryClient } from "@tanstack/react-query";

const queryClient = useQueryClient();

// View all cached queries
console.log(queryClient.getQueryCache().getAll());

// View specific query
const queryKey = ["query:files.directory_listing", libraryId, { path }];
console.log(queryClient.getQueryData(queryKey));

Migration

From useLibraryQuery

// Before (no real-time updates)
const { data } = useLibraryQuery({
	type: "locations.list",
	input: {},
});

// After (instant updates)
const { data } = useNormalizedQuery({
	query: "locations.list",
	input: null,
	resourceType: "location",
});

Backward Compatibility

The old useNormalizedCache name is aliased:

// Both work identically
import { useNormalizedQuery } from "@sd/ts-client";
import { useNormalizedCache } from "@sd/ts-client"; // Alias

// Prefer useNormalizedQuery for new code

Technical Details

Exported Functions

Core logic is exported for testing:

import {
	filterBatchResources, // Filter resources by pathScope
	updateBatchResources, // Update cache with batch
	updateSingleResource, // Update single resource
	deleteResource, // Remove from cache
	safeMerge, // Deep merge utility
	handleResourceEvent, // Event dispatcher
} from "@sd/ts-client/hooks/useNormalizedQuery";

Runtime Dependencies

ts-deepmerge - Type-safe deep merging
valibot - Runtime event validation
tiny-invariant - Assertion helpers
type-fest - TypeScript utilities
@tanstack/react-query - Core caching

Subscription Lifecycle

1. Component mounts
   ↓
2. useNormalizedQuery creates subscription
   ↓
3. Backend creates filtered event stream
   ↓
4. Events flow: Backend → Tauri → Frontend → Hook → Cache
   ↓
5. Component unmounts
   ↓
6. Cleanup function called
   ↓
7. Tauri cancels background task
   ↓
8. Backend receives Unsubscribe
   ↓
9. Unix socket closed
   ↓
10. Connection freed

Common Patterns

List with Real-Time Updates

const { data: items } = useNormalizedQuery({
	query: "items.list",
	input: filters,
	resourceType: "item",
});

// Items list updates instantly when:
// - New items created
// - Existing items modified
// - Items deleted

Directory with Instant File Appearance

const { data: files } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path,
});

// New files appear instantly:
// - Screenshot taken → appears immediately
// - File copied → shows up without refresh
// - File renamed → updates in real-time

Inspector with Sidecar Updates

const { data: file } = useNormalizedQuery({
	query: "files.by_id",
	input: { file_id },
	resourceType: "file",
	resourceId: file_id,
});

// Sidecars update as they're generated:
// - Thumbnail generated → appears instantly
// - Thumbstrip created → shows immediately
// - OCR extracted → updates in real-time

Summary

useNormalizedQuery provides production-grade real-time caching:

Server-side filtering (90%+ event reduction)
Client-side safety (validates and filters)
Proper cleanup (no connection leaks)
Runtime validation (catches bad events)
Type-safe merging (preserves data)
Comprehensive tests (9 Rust + 5 TypeScript)
TanStack Query compatible (all features work)
Cross-device sync (instant updates everywhere)

Use it for any query where data can change and you want instant updates without manual refetching.

​Real-Time Normalized Cache with TanStack Query

​Overview

​Architecture

​Basic Usage

​Directory Listing

​Media View (Recursive)

​Global Resources

​Single Resource Queries

​API Reference

​Options

​Path Filtering Modes

​Exact Mode (Default)

​Recursive Mode

​Server-Side Filtering

​How It Works

​Filter Logic

​Client-Side Safety Filtering

​Event Types

​ResourceChanged (Single)

​ResourceChangedBatch (Multiple)

​ResourceDeleted

​Refresh (Invalidate All)

​Deep Merge Behavior

​No-Merge Fields

​Runtime Validation

​Subscription Multiplexing

​Subscription Cleanup

​Performance

​Event Reduction

​Connection Management

​Testing

​Test Coverage

​Run Tests

​Best Practices

​Always Scope File Queries

​Use Correct Mode for View Type

​Combine with TanStack Query Options

​Advanced Usage

​Content-Addressed Files

​Multiple Instances

​Debugging

​Enable Logging

​Monitor Subscriptions

​Inspect Cache

​Migration

​From useLibraryQuery

​Backward Compatibility

​Technical Details

​Exported Functions

​Runtime Dependencies

​Subscription Lifecycle

​Common Patterns

​List with Real-Time Updates

​Directory with Instant File Appearance

​Inspector with Sidecar Updates

​Summary