seaweedfs

chrislu fa1be8b2b0 perf: add RecordType inference cache to eliminate 37% gateway CPU overhead CRITICAL: Gateway was creating Avro codecs and inferring RecordTypes on EVERY fetch request for schematized topics! Problem (from CPU profile): - NewCodec (Avro): 17.39% CPU (2.35s out of 13.51s) - inferRecordTypeFromAvroSchema: 20.13% CPU (2.72s) - Total schema overhead: 37.52% CPU - Called during EVERY fetch to check if topic is schematized - No caching - recreating expensive goavro.Codec objects repeatedly Root Cause: In the fetch path, isSchematizedTopic() -> matchesSchemaRegistryConvention() -> ensureTopicSchemaFromRegistryCache() -> inferRecordTypeFromCachedSchema() -> inferRecordTypeFromAvroSchema() was being called. The inferRecordTypeFromAvroSchema() function created a NEW Avro decoder (which internally calls goavro.NewCodec()) on every call, even though: 1. The schema.Manager already has a decoder cache by schema ID 2. The same schemas are used repeatedly for the same topics 3. goavro.NewCodec() is expensive (parses JSON, builds schema tree) This was wasteful because: - Same schema string processed repeatedly - No reuse of inferred RecordType structures - Creating codecs just to infer types, then discarding them Solution: Added inferredRecordTypes cache to Handler: Changes to handler.go: - Added inferredRecordTypes map[string]schema_pb.RecordType to Handler - Added inferredRecordTypesMu sync.RWMutex for thread safety - Initialize cache in NewTestHandlerWithMock() and NewSeaweedMQBrokerHandlerWithDefaults() Changes to produce.go: - Added glog import - Modified inferRecordTypeFromAvroSchema(): Check cache first (key: schema string) * Cache HIT: Return immediately (V(4) log) * Cache MISS: Create decoder, infer type, cache result - Modified inferRecordTypeFromProtobufSchema(): * Same caching strategy (key: "protobuf:" + schema) - Modified inferRecordTypeFromJSONSchema(): * Same caching strategy (key: "json:" + schema) Cache Strategy: - Key: Full schema string (unique per schema content) - Value: Inferred *schema_pb.RecordType - Thread-safe with RWMutex (optimized for reads) - No TTL - schemas don't change for a topic - Memory efficient - RecordType is small compared to codec Performance Impact: With 250 fetches/sec across 5 topics (1-3 schemas per topic): - Before: 250 codec creations/sec + 250 inferences/sec = ~5s CPU - After: 3-5 codec creations total (one per schema) = ~0.05s CPU - Reduction: 99% fewer expensive operations Expected CPU Reduction: - Before: 13.51s total, 5.07s schema operations (37.5%) - After: ~8.5s total (-37.5% = 5s saved) - Benefit: 37% lower gateway CPU, more capacity for message processing Cache Consistency: - Schemas are immutable once registered in Schema Registry - If schema changes, schema ID changes, so safe to cache indefinitely - New schemas automatically cached on first use - No need for invalidation or TTL Additional Optimizations: - Protobuf and JSON Schema also cached (same pattern) - Prevents future bottlenecks as more schema formats are used - Consistent caching approach across all schema types Testing: - ✅ Compiles successfully - Ready to deploy and measure CPU improvement under load Priority: HIGH - Eliminates major performance bottleneck in gateway schema path	2 months ago
..
main.go	track messages with testStartTime	2 months ago

chrislu fa1be8b2b0 perf: add RecordType inference cache to eliminate 37% gateway CPU overhead

CRITICAL: Gateway was creating Avro codecs and inferring RecordTypes on
EVERY fetch request for schematized topics!

Problem (from CPU profile):
- NewCodec (Avro): 17.39% CPU (2.35s out of 13.51s)
- inferRecordTypeFromAvroSchema: 20.13% CPU (2.72s)
- Total schema overhead: 37.52% CPU
- Called during EVERY fetch to check if topic is schematized
- No caching - recreating expensive goavro.Codec objects repeatedly

Root Cause:
In the fetch path, isSchematizedTopic() -> matchesSchemaRegistryConvention()
-> ensureTopicSchemaFromRegistryCache() -> inferRecordTypeFromCachedSchema()
-> inferRecordTypeFromAvroSchema() was being called.

The inferRecordTypeFromAvroSchema() function created a NEW Avro decoder
(which internally calls goavro.NewCodec()) on every call, even though:
1. The schema.Manager already has a decoder cache by schema ID
2. The same schemas are used repeatedly for the same topics
3. goavro.NewCodec() is expensive (parses JSON, builds schema tree)

This was wasteful because:
- Same schema string processed repeatedly
- No reuse of inferred RecordType structures
- Creating codecs just to infer types, then discarding them

Solution:
Added inferredRecordTypes cache to Handler:

Changes to handler.go:
- Added inferredRecordTypes map[string]*schema_pb.RecordType to Handler
- Added inferredRecordTypesMu sync.RWMutex for thread safety
- Initialize cache in NewTestHandlerWithMock() and NewSeaweedMQBrokerHandlerWithDefaults()

Changes to produce.go:
- Added glog import
- Modified inferRecordTypeFromAvroSchema():
  * Check cache first (key: schema string)
  * Cache HIT: Return immediately (V(4) log)
  * Cache MISS: Create decoder, infer type, cache result
- Modified inferRecordTypeFromProtobufSchema():
  * Same caching strategy (key: "protobuf:" + schema)
- Modified inferRecordTypeFromJSONSchema():
  * Same caching strategy (key: "json:" + schema)

Cache Strategy:
- Key: Full schema string (unique per schema content)
- Value: Inferred *schema_pb.RecordType
- Thread-safe with RWMutex (optimized for reads)
- No TTL - schemas don't change for a topic
- Memory efficient - RecordType is small compared to codec

Performance Impact:
With 250 fetches/sec across 5 topics (1-3 schemas per topic):
- Before: 250 codec creations/sec + 250 inferences/sec = ~5s CPU
- After: 3-5 codec creations total (one per schema) = ~0.05s CPU
- Reduction: 99% fewer expensive operations

Expected CPU Reduction:
- Before: 13.51s total, 5.07s schema operations (37.5%)
- After: ~8.5s total (-37.5% = 5s saved)
- Benefit: 37% lower gateway CPU, more capacity for message processing

Cache Consistency:
- Schemas are immutable once registered in Schema Registry
- If schema changes, schema ID changes, so safe to cache indefinitely
- New schemas automatically cached on first use
- No need for invalidation or TTL

Additional Optimizations:
- Protobuf and JSON Schema also cached (same pattern)
- Prevents future bottlenecks as more schema formats are used
- Consistent caching approach across all schema types

Testing:
- ✅ Compiles successfully
- Ready to deploy and measure CPU improvement under load

Priority: HIGH - Eliminates major performance bottleneck in gateway schema path

2 months ago

main.go

track messages with testStartTime

2 months ago