Third-Party SDK Monitoring System
Automated change detection with AI agent integration
A minimal-dependency monitoring system that tracks changes in third-party JavaScript SDKs. Uses SHA-256 hashing for change detection, maintains historical archives, and exposes data to AI agents via Model Context Protocol (MCP) for intelligent analysis.
The Problem
Third-party SDKs update without notice, sometimes introducing breaking changes. Teams discovered issues only when customers reported bugs. No systematic way to track SDK changes, understand what changed, or assess breaking change risk.
- SDK updates discovered only after production bugs
- No changelog or release notes from vendor
- Manual checking of SDK source was sporadic and error-prone
- Documentation changes not tracked alongside code changes
Constraints
- Minimal dependencies, can't add monitoring SaaS for one SDK
- Must detect changes in code, docs, and sample repository
- Historical archive for comparing any two versions
- AI agent accessible for intelligent change analysis
The Approach
Zero-dependency core (only MCP SDK for AI integration). SHA-256 hashing for O(1) change detection. Historical archive with full snapshots. MCP server exposes tools for AI agents to search history, compare versions, and identify breaking changes.
Why hash-based detection instead of git diff?
SDK is fetched from CDN, not a git repo. Hash comparison is O(1) regardless of file size. Can detect any change (minified code, docs, samples) with same mechanism. Null hash guards prevent false positives on fetch failures.
Why MCP server for AI agent access?
MCP (Model Context Protocol) is the emerging standard for Claude/Cursor tool integration. Exposes change archive as searchable tools. AI can compare versions, identify breaking changes, generate migration checklists without custom integration.
Why track documentation alongside SDK?
Documentation changes often signal upcoming breaking changes. New pages indicate new features. Removed pages suggest deprecated APIs. Tracking 62+ doc pages provides early warning.
Why separate GitHub Issues for fetch failures vs changes?
Fetch failures shouldn't trigger false 'SDK changed' alerts. Separate issue templates clarify whether action is needed. Failure issues include troubleshooting guidance; change issues include diff details.
Key Tradeoffs
Store full snapshots (not just diffs)
Enables any-to-any version comparison without reconstruction. Storage is cheap. Snapshots organized by source and hash for efficient lookup.
Single external dependency (MCP SDK)
Core monitoring has zero dependencies, pure Node.js 18+ with native fetch. MCP SDK added only for AI agent integration. If MCP removed, core monitoring still works.
Regex-based method extraction has iteration limits
Bounded to 100,000 matches to prevent ReDoS. May miss methods in extremely large SDKs. Acceptable because target SDK is ~223KB.
Implementation Highlights
SHA-256 hash-based change detection
Node crypto module computes content hash. Comparison is hash string equality, O(1) regardless of content size. Null hash handling prevents fetch failures from registering as changes.
Multi-source manifest tracking
SDK (single file), documentation (62+ pages), sample repo (23+ files) tracked independently. Per-page/file hashing enables granular change detection, know exactly which doc page changed.
MCP server with 7 tools
search_changes, compare_versions, get_breaking_changes, generate_migration_checklist, get_history, get_changelog, get_api_schema. AI agents can answer 'what changed since January?' directly.
API impact classification
Extracts SDK methods via regex. Compares method sets between versions. Classifies as BREAKING (removed methods), FEATURE (added methods), or FIX (same methods, different implementation).
GitHub Actions with race condition handling
Daily run at 8 AM UTC. Fetch + rebase before push with 3 retry attempts. Handles concurrent runs gracefully, second run succeeds if first already pushed same state.
Outcomes
What I'd Do Differently
- Add semantic versioning detection. Currently treats all changes equally. Would be useful to detect if SDK itself publishes version numbers in code.
- Build diff visualization UI. CLI changelog is useful but visual diff (like GitHub compare) would help non-technical stakeholders understand changes.
- Implement webhook for immediate notification. Currently GitHub Actions runs daily. Webhook on CDN change would enable near-instant detection for critical SDKs.