From 7c72d66a4b17fdedd072f11a01bd1481e29998d7 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Thu, 12 Nov 2026 22:26:39 -0500 Subject: [PATCH 01/12] feat(codewiki): CU-17tkuw66ma4 read other repositories through the hub's code-graph tools CodeWiki now parses only the repository it documents. Code that repository imports from another repository is looked up on demand by every writing agent through the Flamingo hub's code-graph tools (GET/POST /api/ci/code-documentation/tools, Bearer WEBHOOK_SECRET), served per repository and registered as pydantic_ai tools from the hub's own JSON schemas. A failed call is a ToolFailed result the model sees; no env or an unreadable list means no graph tools. The multi-path support is removed entirely: the --additional-paths option, Config.additional_source_paths, all_source_paths, is_multi_path_mode, the multi-repository parser, Node.is_from_deps, the test-multi-path fixtures, the example and the docs about them. Co-Authored-By: Claude Opus 5.5 --- .test_short_id_normalization.md | 1 - .verify_fqdn_implementation.md | 3 +- CLI_MULTI_PATH_INTEGRATION.md | 338 ------- FQDN_CHANGES_VISUAL.md | 210 ----- FQDN_FIX_SUMMARY.md | 2 +- FQDN_IMPLEMENTATION_SUMMARY.md | 138 --- MULTI_PATH_CHANGES_SUMMARY.md | 244 ----- MULTI_PATH_IMPLEMENTATION.md | 367 -------- MULTI_PATH_LOGGING_EXAMPLES.md | 451 --------- MULTI_PATH_LOGGING_SUMMARY.md | 417 --------- MULTI_PATH_QUICK_REFERENCE.md | 291 ------ SECURITY_ANALYSIS_ADDITIONAL_PATHS.md | 852 ------------------ codewiki/cli/adapters/.doc_generator.md | 4 +- codewiki/cli/adapters/doc_generator.py | 31 +- codewiki/cli/commands/generate.py | 54 +- codewiki/src/.config.md | 16 +- codewiki/src/be/agent_orchestrator.py | 6 +- codewiki/src/be/agent_tools/code_graph.py | 193 ++++ .../generate_sub_module_documentations.py | 5 +- .../src/be/agent_tools/str_replace_editor.py | 9 +- .../src/be/dependency_analyzer/.ast_parser.md | 19 +- .../.dependency_graphs_builder.md | 4 +- .../src/be/dependency_analyzer/ast_parser.py | 276 +----- .../dependency_graphs_builder.py | 15 +- .../be/dependency_analyzer/models/.core.md | 3 +- .../src/be/dependency_analyzer/models/core.py | 3 +- codewiki/src/be/prompt_template.py | 19 + codewiki/src/config.py | 102 +-- docs/README.md | 5 - docs/diagrams/architecture/README-3.mmd | 3 - .../architecture/test-multi-path-core-2.mmd | 5 - .../architecture/test-multi-path-core.mmd | 29 - .../diagrams/architecture/test_fixtures-2.mmd | 18 - .../diagrams/architecture/test_fixtures-3.mmd | 17 - .../diagrams/architecture/test_fixtures-4.mmd | 6 - docs/diagrams/architecture/test_fixtures.mmd | 26 - docs/diagrams/architecture/test_runners-2.mmd | 42 - docs/diagrams/architecture/test_runners-3.mmd | 24 - docs/diagrams/architecture/test_runners.mmd | 30 - docs/reference/architecture/README.md | 5 - .../test-multi-path-core.md | 85 -- .../test-multi-path-core/test_fixtures.md | 135 --- .../test-multi-path-core/test_runners.md | 235 ----- examples/.multi_path_example.md | 42 - examples/multi_path_example.py | 225 ----- pyproject.toml | 3 +- requirements.txt | 4 +- test-multi-path/.debug_parser.md | 55 -- test-multi-path/.integration_test.md | 39 - test-multi-path/.test_multi_path.md | 36 - test-multi-path/INTEGRATION_TEST_RESULTS.md | 185 ---- test-multi-path/README.md | 267 ------ test-multi-path/TEST_RESULTS.md | 326 ------- test-multi-path/debug_parser.py | 49 - test-multi-path/deps/.helper.md | 38 - test-multi-path/deps/.utils.md | 27 - test-multi-path/deps/helper.py | 59 -- test-multi-path/deps/utils.py | 27 - test-multi-path/external/.plugin.md | 42 - test-multi-path/external/plugin.py | 64 -- test-multi-path/integration_test.py | 617 ------------- test-multi-path/main/.controller.md | 37 - test-multi-path/main/.service.md | 51 -- test-multi-path/main/controller.py | 33 - test-multi-path/main/service.py | 60 -- test-multi-path/run_tests.sh | 48 - test-multi-path/test_multi_path.py | 546 ----------- test_short_id_normalization.py | 3 +- verify_fqdn_implementation.py | 6 +- 69 files changed, 282 insertions(+), 7345 deletions(-) delete mode 100644 CLI_MULTI_PATH_INTEGRATION.md delete mode 100644 FQDN_CHANGES_VISUAL.md delete mode 100644 FQDN_IMPLEMENTATION_SUMMARY.md delete mode 100644 MULTI_PATH_CHANGES_SUMMARY.md delete mode 100644 MULTI_PATH_IMPLEMENTATION.md delete mode 100644 MULTI_PATH_LOGGING_EXAMPLES.md delete mode 100644 MULTI_PATH_LOGGING_SUMMARY.md delete mode 100644 MULTI_PATH_QUICK_REFERENCE.md delete mode 100644 SECURITY_ANALYSIS_ADDITIONAL_PATHS.md create mode 100644 codewiki/src/be/agent_tools/code_graph.py delete mode 100644 docs/diagrams/architecture/test-multi-path-core-2.mmd delete mode 100644 docs/diagrams/architecture/test-multi-path-core.mmd delete mode 100644 docs/diagrams/architecture/test_fixtures-2.mmd delete mode 100644 docs/diagrams/architecture/test_fixtures-3.mmd delete mode 100644 docs/diagrams/architecture/test_fixtures-4.mmd delete mode 100644 docs/diagrams/architecture/test_fixtures.mmd delete mode 100644 docs/diagrams/architecture/test_runners-2.mmd delete mode 100644 docs/diagrams/architecture/test_runners-3.mmd delete mode 100644 docs/diagrams/architecture/test_runners.mmd delete mode 100644 docs/reference/architecture/test-multi-path-core/test-multi-path-core.md delete mode 100644 docs/reference/architecture/test-multi-path-core/test_fixtures.md delete mode 100644 docs/reference/architecture/test-multi-path-core/test_runners.md delete mode 100644 examples/.multi_path_example.md delete mode 100644 examples/multi_path_example.py delete mode 100644 test-multi-path/.debug_parser.md delete mode 100644 test-multi-path/.integration_test.md delete mode 100644 test-multi-path/.test_multi_path.md delete mode 100644 test-multi-path/INTEGRATION_TEST_RESULTS.md delete mode 100644 test-multi-path/README.md delete mode 100644 test-multi-path/TEST_RESULTS.md delete mode 100644 test-multi-path/debug_parser.py delete mode 100644 test-multi-path/deps/.helper.md delete mode 100644 test-multi-path/deps/.utils.md delete mode 100644 test-multi-path/deps/helper.py delete mode 100644 test-multi-path/deps/utils.py delete mode 100644 test-multi-path/external/.plugin.md delete mode 100644 test-multi-path/external/plugin.py delete mode 100755 test-multi-path/integration_test.py delete mode 100644 test-multi-path/main/.controller.md delete mode 100644 test-multi-path/main/.service.md delete mode 100644 test-multi-path/main/controller.py delete mode 100644 test-multi-path/main/service.py delete mode 100755 test-multi-path/run_tests.sh delete mode 100755 test-multi-path/test_multi_path.py diff --git a/.test_short_id_normalization.md b/.test_short_id_normalization.md index e8f2acac..1325b77d 100644 --- a/.test_short_id_normalization.md +++ b/.test_short_id_normalization.md @@ -50,6 +50,5 @@ short_to_fqdn = build_short_id_to_fqdn_map(components) ## Notes -- Dependency components (`is_from_deps=True`) are included in the mapping - The normalization priority is: exact FQDN match → short ID lookup → not found - Source: [`test_short_id_normalization.py`](https://github.com/flamingo-stack/CodeWiki/blob/main/test_short_id_normalization.py) \ No newline at end of file diff --git a/.verify_fqdn_implementation.md b/.verify_fqdn_implementation.md index 44e24144..c782e8a5 100644 --- a/.verify_fqdn_implementation.md +++ b/.verify_fqdn_implementation.md @@ -5,7 +5,7 @@ Verification utility that statically checks whether the FQDN (Fully Qualified Do | Function | Description | |---|---| -| `check_node_model()` | Scans `core.py` for required FQDN-related fields: `short_id`, `namespace`, `is_from_deps` | +| `check_node_model()` | Scans `core.py` for required FQDN-related fields: `short_id`, `namespace` | | `check_ast_parser()` | Scans `ast_parser.py` for correct FQDN construction patterns and `Node` instantiation arguments | | `main()` | Orchestrates both checks, prints a summary, and exits with code `0` (pass) or `1` (fail) | @@ -26,7 +26,6 @@ FQDN Implementation Verification ✅ Node Model FQDN Fields Check: ✓ short_id: ✓ namespace: - ✓ is_from_deps: ✅ AST Parser FQDN Implementation Check: ✓ fqdn = f"{namespace}.{original_id}" diff --git a/CLI_MULTI_PATH_INTEGRATION.md b/CLI_MULTI_PATH_INTEGRATION.md deleted file mode 100644 index 73e2b57e..00000000 --- a/CLI_MULTI_PATH_INTEGRATION.md +++ /dev/null @@ -1,338 +0,0 @@ -# CLI Multi-Path Integration Summary - -## Overview -Successfully added `--additional-paths` CLI option and wired it through the entire pipeline to enable multi-path configuration for dependency analysis. - -## Changes Made - -### 1. CLI Command (`codewiki/cli/commands/generate.py`) - -**Added CLI Option:** -```python -@click.option( - "--additional-paths", - type=str, - default=None, - help="Comma-separated additional paths to include in dependency analysis (e.g., 'vendor/deps,external/libs')", -) -``` - -**Added Parameter Validation:** -```python -# Validate and parse additional paths -additional_paths_list = None -if additional_paths: - additional_paths_list = parse_patterns(additional_paths) - - # Validate that paths exist - invalid_paths = [] - for path in additional_paths_list: - full_path = repo_path / path - if not full_path.exists(): - invalid_paths.append(path) - - if invalid_paths: - raise RepositoryError( - f"Additional paths not found:\n" - f" {', '.join(invalid_paths)}\n\n" - f"Please ensure all paths exist relative to repository root:\n" - f" Repository: {repo_path}\n" - f" Invalid paths: {invalid_paths}" - ) -``` - -**Added to Config Dictionary:** -```python -# Additional paths for dependency analysis -'additional_paths': additional_paths_list, -``` - -**Added Usage Examples:** -```bash -# Include additional dependency paths (e.g., vendor, external libs) -$ codewiki generate --additional-paths "vendor/packages,external/deps" -``` - -### 2. CLI Adapter (`codewiki/cli/adapters/doc_generator.py`) - -**Added Type Import:** -```python -from typing import Dict, Any, List -``` - -**Wire Through to Backend:** -```python -# Additional paths for dependency analysis (convert to absolute paths) -additional_source_paths=[ - str((Path(self.repo_path) / path).resolve()) - for path in self.config.get('additional_paths', []) -] if self.config.get('additional_paths') else None -``` - -**Added Verbose Logging:** -```python -if backend_config.additional_source_paths: - print(f" ├─ Additional Paths ({len(backend_config.additional_source_paths)}):") - for path in backend_config.additional_source_paths: - print(f" │ └─ {path}") -``` - -### 3. Dependency Graph Builder (`codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py`) - -**Updated to Use Multi-Path from Config:** -```python -# Get custom include/exclude patterns from config -include_patterns = self.config.include_patterns if self.config.include_patterns else None -exclude_patterns = self.config.exclude_patterns if self.config.exclude_patterns else None - -# Build list of paths to analyze -repo_paths = self.config.all_source_paths # Property that returns List[str] - -# Log multi-path mode if enabled -if self.config.is_multi_path_mode(): - logger.info(f"🔀 Multi-path mode enabled: analyzing {len(repo_paths)} paths") - for i, path in enumerate(repo_paths, 1): - logger.info(f" {i}. {path}") - -parser = DependencyParser( - repo_paths if len(repo_paths) > 1 else repo_paths[0], - include_patterns=include_patterns, - exclude_patterns=exclude_patterns -) -``` - -## CLI Usage Examples - -### Basic Usage -```bash -# Single additional path -codewiki generate --additional-paths "vendor/deps" - -# Multiple paths (comma-separated) -codewiki generate --additional-paths "vendor/deps,external/libs,third_party/packages" -``` - -### Combined with Other Options -```bash -# With verbose output -codewiki generate --additional-paths "vendor,external" --verbose - -# With include/exclude patterns -codewiki generate --additional-paths "vendor" \ - --include "*.py,*.js" \ - --exclude "*test*,*spec*" - -# With focus modules -codewiki generate --additional-paths "deps" \ - --focus "src/core,src/api" - -# Full example -codewiki generate \ - --additional-paths "vendor/packages,external/deps" \ - --include "*.py,*.js" \ - --exclude "*Tests*,*Specs*" \ - --doc-type architecture \ - --verbose -``` - -## Error Handling - -### Path Validation -The CLI performs early validation of all additional paths: - -1. **Parse comma-separated paths** using existing `parse_patterns()` function -2. **Check each path exists** relative to repository root -3. **Raise RepositoryError** with helpful message if any path is invalid: - ``` - Additional paths not found: - vendor/missing, external/nonexistent - - Please ensure all paths exist relative to repository root: - Repository: /path/to/repo - Invalid paths: ['vendor/missing', 'external/nonexistent'] - ``` - -### Backend Validation -Backend Config validates all paths using `validate_source_paths()`: -- Checks path exists -- Checks path is a directory -- Checks read access permissions - -## Integration Points - -### Data Flow -``` -CLI Option (--additional-paths) - ↓ parse_patterns() -CLI Command (additional_paths: str) - ↓ validation + split -CLI Command (additional_paths_list: List[str]) - ↓ pass to generator config -CLIDocumentationGenerator (config['additional_paths']) - ↓ convert to absolute paths -BackendConfig.from_cli(additional_source_paths: List[str]) - ↓ validate + store -DependencyGraphBuilder (config.all_source_paths) - ↓ use in parser -DependencyParser(repo_paths: Union[str, List[str]]) -``` - -### Existing Backend Support -The backend already supported multi-path mode: - -- **Config class** has `additional_source_paths` field -- **Config.all_source_paths** property returns list of all paths -- **Config.is_multi_path_mode()** checks if multi-path enabled -- **Config.validate_source_paths()** validates all paths -- **DependencyParser** accepts `Union[str, List[str]]` for repo_path - -## Testing - -### CLI Help -```bash -$ codewiki generate --help -... - --additional-paths TEXT Comma-separated additional paths to include - in dependency analysis (e.g., - 'vendor/deps,external/libs') -``` - -### Type Checking -```bash -$ mypy codewiki/cli/commands/generate.py -# No type errors -``` - -### Manual Testing -```bash -# Test with valid paths -$ codewiki generate --additional-paths "src,tests" --verbose - -# Test with invalid paths (should fail with clear error) -$ codewiki generate --additional-paths "missing,nonexistent" -``` - -### Test Results - -**Test 1: Invalid Paths (Expected Failure)** -```bash -$ cd /tmp/test_repo -$ codewiki generate --additional-paths "missing,nonexistent" - -✗ Additional paths not found: - missing, nonexistent - -Please ensure all paths exist relative to repository root: - Repository: /private/tmp/test_repo - Invalid paths: ['missing', 'nonexistent'] -``` -✅ **Result**: Early validation catches invalid paths with clear error message - -**Test 2: Valid Multi-Path Configuration** -```bash -$ cd /tmp/test_repo -$ codewiki generate --additional-paths "vendor,external" --verbose - -[1/4] Validating configuration... -✓ Configuration valid -[2/4] Validating repository... -✓ Repository valid: test_repo - Detected languages: Python (2 files) -✓ Output directory: /private/tmp/test_repo/docs -[4/4] Generating documentation... - - Max tokens: 128000 - Max token/module: 36369 - Max token/leaf module: 16000 - Max depth: 2 - Additional paths: ['vendor', 'external'] - -📋 Generation Configuration: - ├─ Cluster Model: gpt-5.2 - │ └─ Base URL: https://api.openai.com/v1 - │ └─ Max Tokens: 128000 - ├─ Main Model: gpt-5.2-chat-latest - │ └─ Base URL: https://api.openai.com/v1 - │ └─ Max Tokens: 128000 - ├─ Fallback Model: claude-opus-4-5-20251101 - │ └─ Base URL: https://api.anthropic.com/v1 - │ └─ Max Tokens: 200000 - ├─ Module Settings: - │ ├─ Max tokens/module: 36369 - │ ├─ Max tokens/leaf: 16000 - │ └─ Max depth: 2 - ├─ Additional Paths (2): - │ └─ /private/tmp/test_repo/vendor - │ └─ /private/tmp/test_repo/external - -[00:00] Phase 1/5: Dependency Analysis -[00:00] Initializing dependency analyzer... - ├─ Repository: /private/tmp/test_repo - └─ Output: /private/tmp/test_repo/docs/temp -[00:00] Scanning repository for source files... -[00:00] Building dependency graph... -INFO 🔀 Multi-path mode enabled: analyzing 3 paths - 1. /private/tmp/test_repo - 2. /private/tmp/test_repo/vendor - 3. /private/tmp/test_repo/external -``` -✅ **Result**: All paths parsed, validated, converted to absolute paths, and passed to analyzer - -## Benefits - -1. **User-Friendly Syntax**: Comma-separated paths (e.g., `path1,path2,path3`) -2. **Early Validation**: Paths checked before backend processing starts -3. **Clear Error Messages**: Helpful feedback if paths don't exist -4. **Verbose Logging**: Shows all paths being analyzed in verbose mode -5. **Backward Compatible**: Optional parameter, no impact on existing usage -6. **Type Safe**: Full type annotations and type checking pass - -## Future Enhancements - -### Optional: Support Repeated Option -Could add support for repeated option syntax: -```bash -# Alternative syntax (requires additional implementation) -codewiki generate --additional-path vendor --additional-path external -``` - -Implementation would change option to: -```python -@click.option( - "--additional-path", - "additional_paths", - multiple=True, - help="Additional path to include (can be specified multiple times)" -) -``` - -Then handle both comma-separated and repeated options. - -## Files Modified - -1. `/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/cli/commands/generate.py` - - Added CLI option - - Added parameter validation - - Added usage examples - - Wired through to generator config - -2. `/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/cli/adapters/doc_generator.py` - - Added type import - - Convert relative paths to absolute - - Pass to backend config - - Added verbose logging - -3. `/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py` - - Use config.get_all_source_paths() - - Log multi-path mode info - - Pass list to DependencyParser - -## Conclusion - -The multi-path configuration is now fully exposed via CLI with: -- ✅ User-friendly comma-separated syntax -- ✅ Early validation with clear error messages -- ✅ Complete integration through all layers -- ✅ Verbose logging support -- ✅ Type safety -- ✅ Backward compatibility diff --git a/FQDN_CHANGES_VISUAL.md b/FQDN_CHANGES_VISUAL.md deleted file mode 100644 index 5bf95bb3..00000000 --- a/FQDN_CHANGES_VISUAL.md +++ /dev/null @@ -1,210 +0,0 @@ -# FQDN Implementation - Visual Changes - -## Node Model Changes - -### BEFORE -```python -class Node(BaseModel): - id: str # Could collide across namespaces - name: str - component_type: str - # ... other fields ... - component_id: Optional[str] = None -``` - -### AFTER -```python -class Node(BaseModel): - id: str # FQDN format: {namespace}.{original_id} - name: str - component_type: str - # ... other fields ... - component_id: Optional[str] = None - - # NEW: FQDN metadata fields (backward compatible) - short_id: str = "" # Original ID for display - namespace: str = "" # Source namespace - is_from_deps: bool = False # Dependency flag -``` - ---- - -## Multi-Path Node Creation Changes - -### BEFORE -```python -def _build_namespaced_components( - self, - call_graph_result: Dict, - namespace: str, - namespace_mapping: Dict[str, str] -) -> Dict[str, Node]: - for func_dict in functions: - original_id = func_dict.get("id", "") - namespaced_id = f"{namespace}.{original_id}" - - node = Node( - id=namespaced_id, - name=func_dict.get("name", ""), - # ... other fields ... - component_id=namespaced_id - ) - - components[namespaced_id] = node # ⚠️ Variable naming inconsistent -``` - -### AFTER -```python -def _build_namespaced_components( - self, - call_graph_result: Dict, - namespace: str, - namespace_mapping: Dict[str, str], - repo_index: int = 0 # ✨ NEW: Track main vs deps -) -> Dict[str, Node]: - for func_dict in functions: - original_id = func_dict.get("id", "") - fqdn = f"{namespace}.{original_id}" # ✨ Clear FQDN variable - - node = Node( - id=fqdn, # ✨ FQDN as primary ID - name=func_dict.get("name", ""), - # ... other fields ... - component_id=fqdn, - # ✨ NEW: FQDN metadata - short_id=original_id, - namespace=namespace, - is_from_deps=(repo_index > 0) - ) - - components[fqdn] = node # ✅ Consistent FQDN usage -``` - ---- - -## Single-Path Node Creation Changes - -### BEFORE -```python -def _build_components_from_analysis(self, call_graph_result: Dict): - for func_dict in functions: - component_id = func_dict.get("id", "") - - node = Node( - id=component_id, # ⚠️ No namespace - name=func_dict.get("name", ""), - # ... other fields ... - component_id=component_id - ) - - self.components[component_id] = node # ⚠️ Potential collisions - - # Modules stored without namespace - if "." in component_id: - module_path = ".".join(component_id.split(".")[:-1]) - self.modules.add(module_path) -``` - -### AFTER -```python -def _build_components_from_analysis(self, call_graph_result: Dict): - # ✨ NEW: Compute namespace from repo path - namespace = self._get_namespace_from_path(self.repo_path) - - for func_dict in functions: - original_id = func_dict.get("id", "") - fqdn = f"{namespace}.{original_id}" # ✨ Construct FQDN - - node = Node( - id=fqdn, # ✨ FQDN as primary ID - name=func_dict.get("name", ""), - # ... other fields ... - component_id=fqdn, - # ✨ NEW: FQDN metadata - short_id=original_id, - namespace=namespace, - is_from_deps=False # Single-path = main repo - ) - - self.components[fqdn] = node # ✅ FQDN key - - # ✨ Modules stored with namespace - if "." in original_id: - module_path = ".".join(original_id.split(".")[:-1]) - self.modules.add(f"{namespace}.{module_path}") -``` - ---- - -## Example Data Transformation - -### BEFORE (Potential ID Collision) -```python -# Main repo component -Node( - id="services.auth.login", - name="login", - component_id="services.auth.login" -) - -# Dependency repo component (COLLISION!) -Node( - id="services.auth.login", # ⚠️ Same ID! - name="login", - component_id="services.auth.login" -) -``` - -### AFTER (No Collision with FQDN) -```python -# Main repo component -Node( - id="openframe-frontend.services.auth.login", # ✅ FQDN - name="login", - component_id="openframe-frontend.services.auth.login", - short_id="services.auth.login", # ✨ Clean display name - namespace="openframe-frontend", - is_from_deps=False -) - -# Dependency repo component (NO COLLISION!) -Node( - id="ui-kit.services.auth.login", # ✅ Different FQDN - name="login", - component_id="ui-kit.services.auth.login", - short_id="services.auth.login", # ✨ Same clean name - namespace="ui-kit", - is_from_deps=True # ✨ Marked as dependency -) -``` - ---- - -## Key Improvements - -| Aspect | Before | After | -|--------|--------|-------| -| **ID Format** | `services.auth.login` | `openframe-frontend.services.auth.login` | -| **Collision Risk** | ⚠️ High (cross-repo) | ✅ None (FQDN guarantee) | -| **Source Tracking** | ❌ Not tracked | ✅ `namespace` + `is_from_deps` | -| **Display Name** | `id` (long) | `short_id` (clean) | -| **Dependency Filtering** | ❌ Not possible | ✅ Use `is_from_deps` flag | -| **Module Paths** | `services.auth` | `openframe-frontend.services.auth` | - ---- - -## Backward Compatibility - -All new fields have defaults, ensuring existing code continues to work: - -```python -# Old code creating Node without new fields -node = Node( - id="some_id", - name="some_name", - component_type="function", - file_path="/path/to/file.py", - relative_path="file.py" -) -# ✅ Works! New fields default to: short_id="", namespace="", is_from_deps=False -``` diff --git a/FQDN_FIX_SUMMARY.md b/FQDN_FIX_SUMMARY.md index 686d42c9..febbb997 100644 --- a/FQDN_FIX_SUMMARY.md +++ b/FQDN_FIX_SUMMARY.md @@ -183,7 +183,7 @@ python -m pytest test_fqdn_normalization.py -v ### Step 3: Integration Test ```bash -# Run full workflow with multi-path +# Run full workflow cd /private/tmp/test-codewiki-fqdn /tmp/test-codewiki-fqdn.sh diff --git a/FQDN_IMPLEMENTATION_SUMMARY.md b/FQDN_IMPLEMENTATION_SUMMARY.md deleted file mode 100644 index b9637fd4..00000000 --- a/FQDN_IMPLEMENTATION_SUMMARY.md +++ /dev/null @@ -1,138 +0,0 @@ -# FQDN Implementation Summary - -## Overview -Successfully implemented FQDN (Fully Qualified Domain Name) fields in the Node model and updated all Node creation points to use FQDN consistently as the primary identifier. - -## Changes Made - -### 1. Node Model Updates (`codewiki/src/be/dependency_analyzer/models/core.py`) - -**Added FQDN metadata fields:** -```python -class Node(BaseModel): - id: str # Now FQDN format: {namespace}.{original_id} - - # ... existing fields ... - - # NEW: FQDN metadata fields (all have defaults for backward compatibility) - short_id: str = "" # Original ID without namespace (for display) - namespace: str = "" # Namespace (e.g., "main", "deps", "ui-kit") - is_from_deps: bool = False # True if from dependencies, False if from main repo -``` - -**Key Points:** -- `id` field is now FQDN (primary key) -- `short_id` stores the original ID for display purposes -- `namespace` tracks which repository/source the component comes from -- `is_from_deps` boolean flag for quick identification of dependency components -- All new fields have defaults for backward compatibility - -### 2. Multi-Path Node Creation (`ast_parser.py:_build_namespaced_components`) - -**Updated signature:** -```python -def _build_namespaced_components( - self, - call_graph_result: Dict, - namespace: str, - namespace_mapping: Dict[str, str], - repo_index: int = 0 # NEW: 0 = main, >0 = deps -) -> Dict[str, Node]: -``` - -**Updated Node creation:** -```python -# Create FQDN (namespaced component ID) -fqdn = f"{namespace}.{original_id}" - -# Create node with FQDN as id and metadata fields -node = Node( - id=fqdn, # FQDN as primary identifier - # ... existing fields ... - component_id=fqdn, - # FQDN metadata fields - short_id=original_id, - namespace=namespace, - is_from_deps=(repo_index > 0) # True for dependencies -) - -# Dictionary key is FQDN -components[fqdn] = node -``` - -**Updated caller:** -```python -# In _parse_multiple_repositories() -for idx, repo_path in enumerate(self.repo_paths): - # ... - repo_components = self._build_namespaced_components( - call_graph_result, - namespace, - namespace_mapping, - repo_index=idx # Pass index: 0=main, >0=deps - ) -``` - -### 3. Single-Path Node Creation (`ast_parser.py:_build_components_from_analysis`) - -**Updated implementation:** -```python -def _build_components_from_analysis(self, call_graph_result: Dict): - # Compute namespace from repo path - namespace = self._get_namespace_from_path(self.repo_path) - - for func_dict in functions: - original_id = func_dict.get("id", "") - - # Construct FQDN: {namespace}.{original_id} - fqdn = f"{namespace}.{original_id}" - - node = Node( - id=fqdn, # FQDN as primary identifier - # ... existing fields ... - component_id=fqdn, - # FQDN metadata fields - short_id=original_id, - namespace=namespace, - is_from_deps=False # Single-path = main repo - ) - - # Dictionary key is FQDN - self.components[fqdn] = node - - # Update mapping for dependency resolution - component_id_mapping[original_id] = fqdn - component_id_mapping[fqdn] = fqdn -``` - -## Key Principles Applied - -1. **`node.id` is ALWAYS FQDN** - Throughout the entire codebase -2. **Dictionary keys use FQDN** - `components[fqdn] = node` -3. **Metadata preserved** - `short_id`, `namespace`, `is_from_deps` for display/filtering -4. **Backward compatibility** - All new fields have defaults -5. **Consistent mapping** - `component_id_mapping` properly handles FQDN conversion - -## Benefits - -1. **No ID Collisions** - FQDNs guarantee unique identifiers across all sources -2. **Source Tracking** - `namespace` and `is_from_deps` enable filtering/grouping -3. **Display Flexibility** - `short_id` provides clean names for UI -4. **Dependency Resolution** - FQDN-based dependencies work across namespaces -5. **Backward Compatible** - Existing code continues to work with defaults - -## Testing Validation - -- ✅ Node model syntax validated -- ✅ ast_parser.py syntax validated -- ✅ All Node creation points updated to use FQDN -- ✅ Dependency resolution uses FQDN mapping -- ✅ Module tracking uses namespaced paths - -## Next Steps - -The FQDN implementation is complete. Future work may include: -1. Update display components to use `short_id` for cleaner UI -2. Add filtering/grouping by `namespace` and `is_from_deps` -3. Enhance cross-namespace dependency visualization -4. Document FQDN format in API documentation diff --git a/MULTI_PATH_CHANGES_SUMMARY.md b/MULTI_PATH_CHANGES_SUMMARY.md deleted file mode 100644 index 3df8cc21..00000000 --- a/MULTI_PATH_CHANGES_SUMMARY.md +++ /dev/null @@ -1,244 +0,0 @@ -# Multi-Path Implementation - Changes Summary - -## Overview -Implementation of multi-path support in DependencyParser and RepoAnalyzer, enabling analysis of multiple source directories with proper component namespacing and cross-repository dependency resolution. - -## Files Modified - -### 1. `/codewiki/src/be/dependency_analyzer/ast_parser.py` -**Lines**: 162 → 373 (+211 lines, +130% growth) - -#### Changes: -- **Constructor (`__init__`)**: Now accepts `Union[str, List[str]]` for repo_path -- **New attribute**: `self.repo_paths: List[str]` - stores all repository paths -- **Maintained**: `self.repo_path` for backward compatibility - -#### New Methods (6): -1. `_parse_single_repository(repo_path, filtered_folders)` - Single path parsing -2. `_parse_multiple_repositories(filtered_folders)` - Multi-path orchestration -3. `_get_namespace_from_path(repo_path)` - Namespace generation -4. `_build_namespaced_components(call_graph_result, namespace, namespace_mapping)` - Component namespacing -5. `_resolve_cross_namespace_dependencies(all_components, namespace_mapping)` - Cross-repo dependency resolution - -#### Modified Methods (1): -1. `parse_repository(filtered_folders)` - Now dispatches to single or multi-path logic - -### 2. `/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py` -**Lines**: 129 → 213 (+84 lines, +65% growth) - -#### Changes: -- **Constructor (`__init__`)**: Added `self._namespace_roots: Dict[str, str]` tracking -- **Updated**: `analyze_repository_structure()` now accepts `Union[str, List[str]]` - -#### New Methods (2): -1. `_analyze_multiple_repositories(repo_dirs)` - Multi-path analysis and merging -2. `_get_namespace_from_path(repo_path)` - Namespace generation (consistent with DependencyParser) - -#### Modified Methods (1): -1. `analyze_repository_structure(repo_dir)` - Handles both single and multi-path modes - -## Documentation Created - -### 1. `MULTI_PATH_IMPLEMENTATION.md` (423 lines) -Complete technical documentation covering: -- Architecture and design decisions -- Component namespacing format -- Dependency resolution strategies -- API changes and backward compatibility -- Usage examples and edge cases -- Performance characteristics -- Testing guidelines -- Migration guide - -### 2. `MULTI_PATH_QUICK_REFERENCE.md` (334 lines) -Quick reference guide with: -- Basic usage patterns -- Component ID format table -- Key methods reference -- Common patterns and recipes -- File pattern examples -- Error handling -- Performance tips -- Troubleshooting - -### 3. `examples/multi_path_example.py` (283 lines) -Working examples demonstrating: -- Single path usage (backward compatible) -- Multiple path usage with namespacing -- RepoAnalyzer multi-path support -- Component detail access -- Cross-namespace dependency analysis -- Saving multi-path analysis to JSON - -## Key Features Implemented - -### ✅ Multi-Path Support -- Parse multiple repositories in a single operation -- List of paths: `["/repo1", "/repo2", "/repo3"]` - -### ✅ Component Namespacing -- Format: `{namespace}.{original_component_id}` -- Namespace derived from directory name -- Example: `ui-kit.components.Button.render` - -### ✅ Dependency Resolution -1. **Intra-namespace**: Direct ID matching within same repository -2. **Cross-namespace**: Name-based matching across repositories -3. **Fallback**: Preserve unresolved IDs for debugging - -### ✅ Backward Compatibility -- Single string path works exactly as before -- `self.repo_path` maintained for legacy code -- No breaking changes to existing functionality - -### ✅ Merged File Trees (RepoAnalyzer) -- Multiple repos merged into single tree structure -- Each repo wrapped with namespace prefix -- Combined summary statistics - -### ✅ Type Safety -- Type hints: `Union[str, List[str]]` -- All methods properly typed -- Compatible with mypy - -## Architecture Decisions - -| Decision | Rationale | -|----------|-----------| -| Namespace from directory name | Simple, predictable, human-readable | -| Backward compatibility preserved | No disruption to existing code | -| Cross-namespace resolution by name | Flexible, handles most real-world cases | -| No custom namespace mapping (yet) | Keep initial implementation simple | -| Maintain `self.repo_path` | Preserve legacy code compatibility | - -## Testing - -### Syntax Validation: ✅ Passed -```bash -python3 -c "compile('ast_parser.py')" # ✅ Valid -python3 -c "compile('repo_analyzer.py')" # ✅ Valid -``` - -### Method Verification: ✅ All Present -- DependencyParser: 9/9 methods ✓ -- RepoAnalyzer: 7/7 methods ✓ - -## Usage Examples - -### Before (Single Path) -```python -parser = DependencyParser("/path/to/repo") -components = parser.parse_repository() -# Component IDs: "src.module.Component" -``` - -### After (Multi-Path) -```python -parser = DependencyParser([ - "/path/to/openframe-frontend", - "/path/to/ui-kit" -]) -components = parser.parse_repository() -# Component IDs: "openframe-frontend.src.module.Component" -# "ui-kit.components.Button" -``` - -## Impact Analysis - -### Backward Compatibility: ✅ 100% -- All existing single-path code works unchanged -- No API breaking changes -- Deprecated: None -- Removed: None - -### Performance Impact -- Single path: No performance change -- Multi-path: O(n * m) where n = repos, m = avg components per repo -- Memory: Linear growth with number of repositories - -### Type Safety: ✅ Enhanced -- Added `Union[str, List[str]]` types -- Maintained type consistency -- No type regressions - -## Migration Path - -### Phase 1: No Changes Required (Current State) -Existing code continues to work: -```python -parser = DependencyParser("/single/path") -``` - -### Phase 2: Opt-In Multi-Path (When Ready) -Update to multi-path when needed: -```python -parser = DependencyParser(["/path1", "/path2"]) -``` - -### Phase 3: Update Consumers (Optional) -Update code that processes component IDs: -```python -# Old: Assumes non-namespaced IDs -comp_id.split('.') # ["src", "module", "Component"] - -# New: Handle namespaced IDs -parts = comp_id.split('.') -namespace = parts[0] # "ui-kit" -module = '.'.join(parts[1:-1]) # "src.module" -name = parts[-1] # "Component" -``` - -## Future Enhancements - -### Potential Additions -1. Custom namespace mapping (override directory name) -2. Component visibility controls (public/private) -3. Namespace aliases (shorter IDs) -4. More sophisticated cross-namespace resolution -5. Performance optimizations for large multi-repo setups -6. Namespace-specific include/exclude patterns - -### Not Implemented (By Design) -- Breaking changes to single-path mode -- Required migration for existing code -- Complex configuration files -- Automatic namespace inference from imports - -## Validation Checklist - -- [x] Syntax validation passed -- [x] All methods implemented -- [x] Type hints added -- [x] Backward compatibility maintained -- [x] Documentation complete -- [x] Examples provided -- [x] No breaking changes -- [x] Code compiles successfully -- [x] Method signatures correct -- [x] Error handling preserved - -## Implementation Statistics - -| Metric | Value | -|--------|-------| -| Files modified | 2 | -| Lines added | +295 | -| New methods | 8 | -| Modified methods | 2 | -| Documentation pages | 3 | -| Code examples | 5 | -| Test coverage | Manual validation | -| Breaking changes | 0 | - -## Summary - -Successfully implemented multi-path support with: -- ✅ Zero breaking changes -- ✅ Complete backward compatibility -- ✅ Comprehensive documentation -- ✅ Working examples -- ✅ Type safety -- ✅ Proper namespacing -- ✅ Cross-repository dependency resolution - -The implementation is production-ready and can be merged without impacting existing functionality. diff --git a/MULTI_PATH_IMPLEMENTATION.md b/MULTI_PATH_IMPLEMENTATION.md deleted file mode 100644 index 5c2201a7..00000000 --- a/MULTI_PATH_IMPLEMENTATION.md +++ /dev/null @@ -1,367 +0,0 @@ -# Multi-Path Support Implementation - -## Overview - -This document describes the implementation of multi-path support in `DependencyParser` and `RepoAnalyzer`, allowing the analysis of multiple source directories with proper component namespacing and cross-repository dependency resolution. - -## Architecture - -### Component Namespacing - -When parsing multiple repositories, each component is namespaced to prevent ID collisions: - -``` -Format: {namespace}.{original_component_id} - -Example: - Single path: "src.components.Button.render" - Multi-path: "openframe-frontend.src.components.Button.render" -``` - -The namespace is derived from the repository directory name (e.g., `openframe-frontend`, `ui-kit`). - -### Dependency Resolution - -Dependencies are resolved in three stages: - -1. **Intra-namespace**: Dependencies within the same repository use namespaced IDs -2. **Cross-namespace**: Dependencies between repositories are resolved by component name matching -3. **Fallback**: Unresolved dependencies retain their original ID for debugging - -## DependencyParser Changes - -### Constructor (`__init__`) - -**Before:** -```python -def __init__(self, repo_path: str, include_patterns: List[str] = None, exclude_patterns: List[str] = None): - self.repo_path = os.path.abspath(repo_path) -``` - -**After:** -```python -def __init__(self, repo_path: Union[str, List[str]], include_patterns: List[str] = None, exclude_patterns: List[str] = None): - if isinstance(repo_path, str): - self.repo_paths = [os.path.abspath(repo_path)] - self.repo_path = self.repo_paths[0] # Backward compatibility - else: - self.repo_paths = [os.path.abspath(p) for p in repo_path] - self.repo_path = self.repo_paths[0] -``` - -**Backward Compatibility:** -- Single string path: Works exactly as before -- List of paths: New multi-path functionality -- `self.repo_path` maintained for legacy code - -### New Methods - -#### `_parse_single_repository(repo_path, filtered_folders)` -- Extracts original `parse_repository` logic -- Maintains backward compatibility -- No changes to component IDs - -#### `_parse_multiple_repositories(filtered_folders)` -- Orchestrates parsing of multiple repositories -- Creates namespace for each repository -- Merges components with namespaced IDs -- Resolves cross-namespace dependencies - -#### `_get_namespace_from_path(repo_path)` -- Generates namespace from directory name -- Example: `/path/to/ui-kit` → `"ui-kit"` - -#### `_build_namespaced_components(call_graph_result, namespace, namespace_mapping)` -- Builds components with namespace prefix -- Maintains mapping of original → namespaced IDs -- Tracks modules with namespaced paths - -#### `_resolve_cross_namespace_dependencies(all_components, namespace_mapping)` -- Resolves dependencies across repository boundaries -- Uses name matching for cross-namespace resolution -- Preserves unresolved dependencies for debugging - -### Updated Method - -#### `parse_repository(filtered_folders)` -- Now dispatches to single or multi-path parsing -- Returns unified component dictionary - -## RepoAnalyzer Changes - -### Constructor (`__init__`) - -**Added:** -```python -self._namespace_roots: Dict[str, str] = {} # namespace -> repo_path -``` - -Tracks which namespace corresponds to which repository path. - -### Updated Methods - -#### `analyze_repository_structure(repo_dir)` - -**Before:** -```python -def analyze_repository_structure(self, repo_dir: str) -> Dict: - file_tree = self._build_file_tree(repo_dir) - return { - "file_tree": file_tree, - "summary": {...} - } -``` - -**After:** -```python -def analyze_repository_structure(self, repo_dir: Union[str, List[str]]) -> Dict: - if isinstance(repo_dir, str): - # Single path - backward compatible - return self._analyze_single_repository(repo_dir) - else: - # Multiple paths - return self._analyze_multiple_repositories(repo_dir) -``` - -### New Methods - -#### `_analyze_multiple_repositories(repo_dirs)` -- Analyzes each repository separately -- Wraps each tree with namespace prefix -- Merges trees into single structure -- Combines summary statistics - -#### `_get_namespace_from_path(repo_path)` -- Same as DependencyParser implementation -- Ensures consistent namespacing - -## Usage Examples - -### Single Path (Backward Compatible) - -```python -from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser - -# Old code continues to work -parser = DependencyParser("/path/to/repo") -components = parser.parse_repository() - -# Component IDs unchanged -# "src.utils.helpers.formatDate" -``` - -### Multiple Paths - -```python -from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser - -# New multi-path support -paths = [ - "/path/to/openframe-frontend", - "/path/to/ui-kit", - "/path/to/shared-lib" -] - -parser = DependencyParser(paths, include_patterns=["*.py", "*.ts"]) -components = parser.parse_repository() - -# Component IDs are namespaced -# "openframe-frontend.src.utils.helpers.formatDate" -# "ui-kit.components.Button.render" -# "shared-lib.utils.api.fetchData" -``` - -### With RepoAnalyzer - -```python -from codewiki.src.be.dependency_analyzer.analysis.repo_analyzer import RepoAnalyzer - -analyzer = RepoAnalyzer( - include_patterns=["*.py", "*.ts"], - exclude_patterns=["*test*", "*spec*"] -) - -# Single path -result = analyzer.analyze_repository_structure("/path/to/repo") - -# Multiple paths -result = analyzer.analyze_repository_structure([ - "/path/to/repo1", - "/path/to/repo2" -]) - -# File tree structure -{ - "file_tree": { - "type": "directory", - "name": "multi-repo", - "children": [ - { - "type": "directory", - "name": "repo1", - "_namespace": "repo1", - "children": [...] - }, - { - "type": "directory", - "name": "repo2", - "_namespace": "repo2", - "children": [...] - } - ] - }, - "summary": { - "total_files": 150, - "total_size_kb": 2048.5, - "repositories": 2, - "namespaces": ["repo1", "repo2"] - } -} -``` - -## Component ID Format - -### Single Path Mode -``` -{module_path}.{component_name} - -Examples: -- src.components.Button.render -- lib.utils.helpers.formatDate -- services.api.UserService.getUser -``` - -### Multi-Path Mode -``` -{namespace}.{module_path}.{component_name} - -Examples: -- openframe-frontend.src.components.Button.render -- ui-kit.lib.utils.helpers.formatDate -- shared-lib.services.api.UserService.getUser -``` - -## Cross-Namespace Dependencies - -When a component in one namespace depends on a component in another: - -```python -# Component in openframe-frontend -{ - "id": "openframe-frontend.src.App.main", - "depends_on": [ - "openframe-frontend.src.utils.logger", # Same namespace - "ui-kit.components.Button" # Cross-namespace - ] -} -``` - -Resolution process: -1. Check if dependency ID exists in `all_components` -2. If not found, extract component name and search by name across namespaces -3. If still not found, keep original ID for debugging - -## Data Structures - -### DependencyParser - -```python -class DependencyParser: - repo_paths: List[str] # All repository paths - repo_path: str # First path (backward compat) - components: Dict[str, Node] # All components (namespaced in multi-path) - modules: Set[str] # All module paths (namespaced in multi-path) - include_patterns: List[str] # File include patterns - exclude_patterns: List[str] # File exclude patterns - analysis_service: AnalysisService -``` - -### RepoAnalyzer - -```python -class RepoAnalyzer: - include_patterns: List[str] - exclude_patterns: List[str] - _namespace_roots: Dict[str, str] # Maps namespace to repo_path -``` - -## Edge Cases Handled - -1. **Empty repositories**: Skipped in multi-path mode -2. **Duplicate namespaces**: Last one wins (deterministic) -3. **Circular dependencies**: Preserved as-is, no cycle detection -4. **Missing dependencies**: Kept with original ID for debugging -5. **Relative paths**: Maintained relative to original repository - -## Performance Characteristics - -- **Time Complexity**: O(n * m) where n = number of repos, m = avg components per repo -- **Space Complexity**: O(n * m) for component storage -- **Namespace Mapping**: O(k) where k = total unique component IDs - -## Testing - -### Test Single Path Compatibility -```python -parser = DependencyParser("/path/to/repo") -assert len(parser.repo_paths) == 1 -assert parser.repo_path == parser.repo_paths[0] -``` - -### Test Multiple Paths -```python -paths = ["/repo1", "/repo2"] -parser = DependencyParser(paths) -assert len(parser.repo_paths) == 2 -components = parser.parse_repository() -# Check namespaced IDs -assert any("repo1." in comp_id for comp_id in components.keys()) -assert any("repo2." in comp_id for comp_id in components.keys()) -``` - -### Test Namespace Generation -```python -parser = DependencyParser(["/path/to/openframe-frontend"]) -namespace = parser._get_namespace_from_path("/path/to/openframe-frontend") -assert namespace == "openframe-frontend" -``` - -## Migration Guide - -### Existing Code -```python -# No changes needed - works as before -parser = DependencyParser(repo_path) -components = parser.parse_repository() -``` - -### New Multi-Path Code -```python -# Update to use list of paths -parser = DependencyParser([repo_path1, repo_path2]) -components = parser.parse_repository() - -# Component IDs now include namespace prefix -for comp_id, component in components.items(): - print(f"Component: {comp_id}") # e.g., "ui-kit.components.Button" -``` - -## Implementation Files - -- `/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/be/dependency_analyzer/ast_parser.py` (355 lines) -- `/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py` (178 lines) - -## Key Decisions - -1. **Namespace from directory name**: Simple, predictable, human-readable -2. **Backward compatibility preserved**: Single string path works unchanged -3. **Cross-namespace resolution by name**: Flexible but may match wrong component in ambiguous cases -4. **No breaking changes**: All existing code continues to work - -## Future Enhancements - -1. Custom namespace mapping (instead of directory name) -2. More sophisticated cross-namespace dependency resolution -3. Namespace aliases for shorter IDs -4. Component visibility controls (public/private across namespaces) -5. Performance optimization for very large multi-repo setups diff --git a/MULTI_PATH_LOGGING_EXAMPLES.md b/MULTI_PATH_LOGGING_EXAMPLES.md deleted file mode 100644 index b7bd9ea5..00000000 --- a/MULTI_PATH_LOGGING_EXAMPLES.md +++ /dev/null @@ -1,451 +0,0 @@ -# Multi-Path Implementation - Example Log Outputs - -## Example 1: OpenFrame Frontend + UI-Kit (2 Paths) - -### Command -```bash -codewiki generate \ - --repo-path=/Users/michaelassraf/Documents/GitHub/openframe-oss-tenant/openframe/services/openframe-frontend/multi-platform-hub \ - --output-dir=output \ - --additional-paths=ui-kit \ - --verbose -``` - -### Expected Log Output - -``` -🔍 CLI received 1 additional path(s): - ├─ Path #1: ui-kit - │ └─ Normalized: /Users/michaelassraf/Documents/GitHub/openframe-oss-tenant/openframe/services/openframe-frontend/multi-platform-hub/ui-kit - └─ Normalized 1 additional path(s) - -📋 Generation Configuration: - ├─ Cluster Model: claude-sonnet-4 - ├─ Main Model: claude-sonnet-4 - ├─ Fallback Model: gpt-4o-mini - ├─ Module Settings: - │ ├─ Max tokens/module: 36369 - │ ├─ Max tokens/leaf: 16000 - │ └─ Max depth: 2 - ├─ Additional Paths (1): - │ └─ /Users/michaelassraf/Documents/GitHub/openframe-oss-tenant/openframe/services/openframe-frontend/multi-platform-hub/ui-kit - └─ Agent Instructions: Configured - -🔍 Validating source paths... - ├─ ✓ Primary path valid: /Users/.../multi-platform-hub - ├─ Validating 1 additional path(s)... - │ ├─ ✓ Additional path #1 valid: /Users/.../ui-kit - └─ ✓ All 1 additional path(s) validated - -===== Stage 1/5: Dependency Analysis ===== - -🔍 Multi-path mode enabled: analyzing 2 source paths - ├─ Primary: /Users/.../multi-platform-hub - └─ Additional #1: /Users/.../ui-kit - -🔍 Analyzing 2 repository paths... - ├─ [1/2] Building file tree for: /Users/.../multi-platform-hub - │ └─ Namespace: 'multi-platform-hub' - │ └─ Found 487 files (8542.35 KB) - ├─ [2/2] Building file tree for: /Users/.../ui-kit - │ └─ Namespace: 'ui-kit' - │ └─ Found 328 files (5123.89 KB) - └─ ✓ Merged 2 repositories: - ├─ Total files: 815 - ├─ Total size: 13666.24 KB - └─ Namespaces: multi-platform-hub, ui-kit - -🔍 Parsing repository files... - -🔍 Multi-path mode detected: analyzing 2 source directories - ├─ Primary: /Users/.../multi-platform-hub - └─ Additional #1: /Users/.../ui-kit - -📂 Analyzing path 1/2: /Users/.../multi-platform-hub - └─ Namespace: 'multi-platform-hub' -📊 Namespace 'multi-platform-hub': found 1243 components - └─ Component types: - • function: 789 - • class: 234 - • interface: 156 - • method: 64 - -📂 Analyzing path 2/2: /Users/.../ui-kit - └─ Namespace: 'ui-kit' -📊 Namespace 'ui-kit': found 856 components - └─ Component types: - • function: 512 - • class: 198 - • interface: 123 - • method: 23 - -🔗 Resolving cross-namespace dependencies... - ├─ Cross-namespace dependency: multi-platform-hub.app.components.Button → ui-kit.components.ui.Button - ├─ Cross-namespace dependency: multi-platform-hub.app.hooks.useToast → ui-kit.hooks.useToast - ├─ Cross-namespace dependency: multi-platform-hub.lib.utils.cn → ui-kit.utils.cn - ... (12 more) - └─ ✓ Resolved 15 cross-namespace dependencies - -📊 Multi-path analysis complete: - ├─ Total components: 2099 - ├─ Total modules: 45 - └─ Namespaces: 2 - - └─ Parsed 2099 components total - └─ Component types found: - • function: 1301 - • class: 432 - • interface: 279 - • method: 87 - - └─ Saved dependency graph to: output/temp/dependency_graphs/multi-platform-hub_dependency_graph.json - -🌿 Filtering leaf nodes (total: 2099)... - └─ Valid types for this codebase: class, interface - -📊 Leaf node filtering complete: - ├─ Kept: 711 nodes - ├─ Skipped (invalid identifier): 0 - ├─ Skipped (wrong type): 1301 - └─ Skipped (not found): 87 - - ├─ Total components: 2099 - ├─ Leaf nodes: 711 - └─ Files analyzed: 815 - -✓ Stage 1/5 complete (45.2s) - -===== Stage 2/5: Module Clustering ===== - - ├─ Leaf nodes to cluster: 711 - ├─ Using model: claude-sonnet-4 - └─ Calling LLM for clustering... - - ├─ Total modules: 18 - └─ Module names: auth, dashboard, devices, logs, admin ... (13 more) - -✓ Stage 2/5 complete (23.7s) -``` - ---- - -## Example 2: Single Path (Backward Compatibility) - -### Command -```bash -codewiki generate \ - --repo-path=/Users/michaelassraf/Documents/GitHub/CodeWiki \ - --output-dir=output \ - --verbose -``` - -### Expected Log Output - -``` -📋 Generation Configuration: - ├─ Cluster Model: claude-sonnet-4 - ├─ Main Model: claude-sonnet-4 - ├─ Fallback Model: gpt-4o-mini - ├─ Module Settings: - │ ├─ Max tokens/module: 36369 - │ ├─ Max tokens/leaf: 16000 - │ └─ Max depth: 2 - -🔍 Validating source paths... - ├─ ✓ Primary path valid: /Users/.../CodeWiki - └─ No additional paths to validate (single-path mode) - -===== Stage 1/5: Dependency Analysis ===== - -📁 Single-path mode: analyzing /Users/.../CodeWiki - - ├─ Repository: /Users/.../CodeWiki - └─ Output: output/temp - -🔍 Parsing repository files... - └─ Parsed 234 components total - └─ Component types found: - • function: 145 - • class: 67 - • method: 22 - - └─ Saved dependency graph to: output/temp/dependency_graphs/CodeWiki_dependency_graph.json - -🌿 Filtering leaf nodes (total: 234)... - └─ Valid types for this codebase: class, function - -📊 Leaf node filtering complete: - ├─ Kept: 212 nodes - ├─ Skipped (invalid identifier): 0 - ├─ Skipped (wrong type): 22 - └─ Skipped (not found): 0 - - ├─ Total components: 234 - ├─ Leaf nodes: 212 - └─ Files analyzed: 156 - -✓ Stage 1/5 complete (12.3s) -``` - ---- - -## Example 3: Three Paths (Complex Multi-Repo) - -### Command -```bash -codewiki generate \ - --repo-path=/app/backend \ - --output-dir=output \ - --additional-paths=../shared-lib,../ui-components \ - --verbose -``` - -### Expected Log Output - -``` -🔍 CLI received 2 additional path(s): - ├─ Path #1: ../shared-lib - │ └─ Normalized: /app/shared-lib - ├─ Path #2: ../ui-components - │ └─ Normalized: /app/ui-components - └─ Normalized 2 additional path(s) - -🔍 Validating source paths... - ├─ ✓ Primary path valid: /app/backend - ├─ Validating 2 additional path(s)... - │ ├─ ✓ Additional path #1 valid: /app/shared-lib - │ ├─ ✓ Additional path #2 valid: /app/ui-components - └─ ✓ All 2 additional path(s) validated - -🔍 Multi-path mode enabled: analyzing 3 source paths - ├─ Primary: /app/backend - ├─ Additional #1: /app/shared-lib - └─ Additional #2: /app/ui-components - -🔍 Analyzing 3 repository paths... - ├─ [1/3] Building file tree for: /app/backend - │ └─ Namespace: 'backend' - │ └─ Found 312 files (5623.12 KB) - ├─ [2/3] Building file tree for: /app/shared-lib - │ └─ Namespace: 'shared-lib' - │ └─ Found 89 files (1234.56 KB) - ├─ [3/3] Building file tree for: /app/ui-components - │ └─ Namespace: 'ui-components' - │ └─ Found 156 files (2345.67 KB) - └─ ✓ Merged 3 repositories: - ├─ Total files: 557 - ├─ Total size: 9203.35 KB - └─ Namespaces: backend, shared-lib, ui-components - -🔍 Multi-path mode detected: analyzing 3 source directories - ├─ Primary: /app/backend - ├─ Additional #1: /app/shared-lib - └─ Additional #2: /app/ui-components - -📂 Analyzing path 1/3: /app/backend - └─ Namespace: 'backend' -📊 Namespace 'backend': found 456 components - └─ Component types: - • class: 234 - • function: 145 - • interface: 67 - • method: 10 - -📂 Analyzing path 2/3: /app/shared-lib - └─ Namespace: 'shared-lib' -📊 Namespace 'shared-lib': found 123 components - └─ Component types: - • function: 78 - • class: 34 - • interface: 11 - -📂 Analyzing path 3/3: /app/ui-components - └─ Namespace: 'ui-components' -📊 Namespace 'ui-components': found 234 components - └─ Component types: - • function: 156 - • class: 67 - • interface: 11 - -🔗 Resolving cross-namespace dependencies... - ├─ Cross-namespace dependency: backend.api.UserController → shared-lib.models.User - ├─ Cross-namespace dependency: backend.api.AuthController → shared-lib.utils.jwt - ├─ Cross-namespace dependency: ui-components.Button → shared-lib.themes.colors - ├─ Cross-namespace dependency: ui-components.Modal → shared-lib.utils.portal - ... (18 more) - └─ ✓ Resolved 22 cross-namespace dependencies - -📊 Multi-path analysis complete: - ├─ Total components: 813 - ├─ Total modules: 28 - └─ Namespaces: 3 -``` - ---- - -## Example 4: Error Case - Invalid Path - -### Command -```bash -codewiki generate \ - --repo-path=/app/backend \ - --output-dir=output \ - --additional-paths=../nonexistent \ - --verbose -``` - -### Expected Log Output - -``` -🔍 CLI received 1 additional path(s): - ├─ Path #1: ../nonexistent - │ └─ Normalized: /app/nonexistent - └─ Normalized 1 additional path(s) - -🔍 Validating source paths... - ├─ ✓ Primary path valid: /app/backend - ├─ Validating 1 additional path(s)... - │ └─ Checking additional path #1: /app/nonexistent - -ERROR: Additional source path #1 does not exist: /app/nonexistent - -Validation failed. Please check your path configuration. -``` - ---- - -## Example 5: Warning - No Files Found - -### Command -```bash -codewiki generate \ - --repo-path=/app/backend \ - --output-dir=output \ - --additional-paths=../empty-dir \ - --include-patterns="*.py" \ - --verbose -``` - -### Expected Log Output - -``` -🔍 Analyzing 2 repository paths... - ├─ [1/2] Building file tree for: /app/backend - │ └─ Namespace: 'backend' - │ └─ Found 312 files (5623.12 KB) - ├─ [2/2] Building file tree for: /app/empty-dir - │ └─ Namespace: 'empty-dir' - │ └─ ⚠️ No files found matching patterns - └─ ✓ Merged 2 repositories: - ├─ Total files: 312 - ├─ Total size: 5623.12 KB - └─ Namespaces: backend, empty-dir - -⚠️ WARNING: Namespace 'empty-dir' has no components -``` - ---- - -## Debugging Tips - -### Enable Verbose Logging -```bash -# Show all DEBUG level logs -codewiki generate --repo-path=. --output-dir=output --verbose - -# Show all logs including component details -export CODEWIKI_LOG_LEVEL=DEBUG -codewiki generate --repo-path=. --output-dir=output -``` - -### Filter Logs by Layer -```bash -# Show only multi-path related logs -codewiki generate ... 2>&1 | grep "🔍\|📂\|📊\|🔗" - -# Show only errors and warnings -codewiki generate ... 2>&1 | grep "ERROR\|WARNING\|⚠️" - -# Show only namespace information -codewiki generate ... 2>&1 | grep "Namespace" -``` - -### Capture Logs to File -```bash -# Full log capture -codewiki generate --repo-path=. --output-dir=output --verbose 2>&1 | tee codewiki.log - -# Time-stamped logs -codewiki generate ... 2>&1 | ts '[%Y-%m-%d %H:%M:%S]' | tee codewiki.log -``` - ---- - -## Key Log Patterns to Look For - -### Success Pattern -``` -✓ Primary path valid -✓ All X additional path(s) validated -✓ Merged X repositories -✓ Resolved X cross-namespace dependencies -📊 Multi-path analysis complete -``` - -### Progress Pattern -``` -[1/3] Building file tree -[2/3] Building file tree -[3/3] Building file tree -📂 Analyzing path 1/3 -📂 Analyzing path 2/3 -📂 Analyzing path 3/3 -``` - -### Error Pattern -``` -ERROR: Additional source path #X does not exist -ERROR: Additional source path #X is not a directory -ERROR: Additional source path #X is not readable -``` - -### Warning Pattern -``` -⚠️ No files found matching patterns -⚠️ Namespace 'X' has no components -⚠️ Leaf node 'X' not found in components -``` - ---- - -## Performance Monitoring - -### Time Tracking -Each log line includes implicit timing through stage progress: -``` -===== Stage 1/5: Dependency Analysis ===== -... (operations) -✓ Stage 1/5 complete (45.2s) - -===== Stage 2/5: Module Clustering ===== -... (operations) -✓ Stage 2/5 complete (23.7s) -``` - -### Component Counting -Track component growth across namespaces: -``` -📊 Namespace 'primary': found 1243 components -📊 Namespace 'ui-kit': found 856 components -📊 Multi-path analysis complete: - ├─ Total components: 2099 ← Sum of all namespaces -``` - -### Memory Usage -Monitor file counts and sizes: -``` - │ └─ Found 487 files (8542.35 KB) ← Per namespace - └─ ✓ Merged 2 repositories: - ├─ Total files: 815 ← Combined - ├─ Total size: 13666.24 KB ← Combined -``` diff --git a/MULTI_PATH_LOGGING_SUMMARY.md b/MULTI_PATH_LOGGING_SUMMARY.md deleted file mode 100644 index a26a329a..00000000 --- a/MULTI_PATH_LOGGING_SUMMARY.md +++ /dev/null @@ -1,417 +0,0 @@ -# Multi-Path Implementation - Comprehensive Logging Summary - -## Overview -Added extensive logging throughout the multi-path implementation to provide complete visibility into the multi-source analysis pipeline. - -## Logging Locations by Layer - -### 1. Config Layer (`codewiki/src/config.py`) - -#### `all_source_paths` Property (Lines 132-145) -**Added Logging:** -```python -logger.debug(f"🔍 all_source_paths property accessed:") -logger.debug(f" ├─ Primary: {paths[0]}") -for i, path in enumerate(paths[1:], 1): - logger.debug(f" └─ Additional #{i}: {path}") -# OR (single-path mode) -logger.debug(f"🔍 all_source_paths property accessed (single-path mode): {paths[0]}") -``` - -**What It Shows:** -- When the property is accessed -- Primary repository path -- All additional paths with numbering -- Whether in single-path or multi-path mode - ---- - -#### `validate_source_paths()` Method (Lines 146-177) -**Added Logging:** -```python -logger.info("🔍 Validating source paths...") -logger.debug(f" ├─ Checking primary path: {self.repo_path}") -logger.info(f" ├─ ✓ Primary path valid: {self.repo_path}") - -logger.info(f" ├─ Validating {len(self.additional_source_paths)} additional path(s)...") -logger.debug(f" │ └─ Checking additional path #{i}: {path}") -logger.info(f" │ ├─ ✓ Additional path #{i} valid: {path}") -logger.info(f" └─ ✓ All {len(self.additional_source_paths)} additional path(s) validated") -# OR (single-path mode) -logger.debug(f" └─ No additional paths to validate (single-path mode)") -``` - -**What It Shows:** -- Validation start -- Each path being checked -- Validation success for each path -- Total count of validated paths -- Errors if paths don't exist or aren't accessible - ---- - -#### `is_multi_path_mode()` Method (Lines 172-184) -**Added Logging:** -```python -logger.debug(f"🔍 Multi-path mode: ENABLED ({len(self.additional_source_paths)} additional paths)") -# OR -logger.debug("🔍 Multi-path mode: DISABLED (single-path)") -``` - -**What It Shows:** -- Whether multi-path mode is enabled -- Count of additional paths when enabled - ---- - -### 2. Parser Layer (`codewiki/src/be/dependency_analyzer/ast_parser.py`) - -#### `_parse_multiple_repositories()` Method (Lines 89-146) -**Added Logging:** -```python -logger.info(f"🔍 Multi-path mode detected: analyzing {len(self.repo_paths)} source directories") -logger.info(f" ├─ Primary: {self.repo_paths[0]}") -for i, path in enumerate(self.repo_paths[1:], 1): - logger.info(f" └─ Additional #{i}: {path}") - -# For each repository: -logger.info(f"\n📂 Analyzing path {idx}/{len(self.repo_paths)}: {repo_path}") -logger.info(f" └─ Namespace: '{namespace}'") - -logger.info(f"📊 Namespace '{namespace}': found {len(repo_components)} components") -logger.info(f" └─ Component types:") -for comp_type, count in sorted(type_counts.items(), key=lambda x: x[1], reverse=True): - logger.info(f" • {comp_type}: {count}") - -logger.info(f"\n🔗 Resolving cross-namespace dependencies...") -logger.info(f" └─ ✓ Resolved {cross_deps_count} cross-namespace dependencies") -# OR -logger.info(f" └─ No cross-namespace dependencies found") - -logger.info(f"\n📊 Multi-path analysis complete:") -logger.info(f" ├─ Total components: {len(all_components)}") -logger.info(f" ├─ Total modules: {len(self.modules)}") -logger.info(f" └─ Namespaces: {len(self.repo_paths)}") -``` - -**What It Shows:** -- Multi-path detection -- List of all paths being analyzed -- Current path being processed with progress (1/3, 2/3, etc.) -- Namespace assignment for each path -- Component count per namespace -- Component type breakdown per namespace -- Cross-namespace dependency resolution -- Final summary statistics - ---- - -#### `_resolve_cross_namespace_dependencies()` Method (Lines 239-273) -**Added Logging:** -```python -logger.debug(f" ├─ Cross-namespace dependency: {component_id} → {other_id}") -``` - -**What It Shows:** -- Each cross-namespace dependency discovered -- Source and target component IDs with namespaces -- Returns total count of cross-namespace dependencies - ---- - -### 3. Analyzer Layer (`codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py`) - -#### `_analyze_multiple_repositories()` Method (Lines 60-114) -**Added Logging:** -```python -logger.info(f"🔍 Analyzing {len(repo_dirs)} repository paths...") - -# For each repository: -logger.info(f" ├─ [{idx}/{len(repo_dirs)}] Building file tree for: {repo_dir}") -logger.info(f" │ └─ Namespace: '{namespace}'") - -logger.info(f" │ └─ Found {files} files ({size:.2f} KB)") -# OR (if no files found) -logger.warning(f" │ └─ ⚠️ No files found matching patterns") - -logger.info(f" └─ ✓ Merged {len(repo_dirs)} repositories:") -logger.info(f" ├─ Total files: {total_files}") -logger.info(f" ├─ Total size: {total_size_kb:.2f} KB") -logger.info(f" └─ Namespaces: {', '.join(self._namespace_roots.keys())}") -``` - -**What It Shows:** -- Total number of repository paths -- Current path being analyzed with progress -- Namespace generation for each path -- File count and size for each path -- Warning if no files found -- Final merged statistics - ---- - -### 4. CLI Layer (`codewiki/cli/adapters/doc_generator.py`) - -#### `generate()` Method (Lines 140-187) -**Added Logging:** -```python -logger.info(f"🔍 CLI received {len(additional_paths_raw)} additional path(s):") - -for i, path in enumerate(additional_paths_raw, 1): - normalized = str((Path(self.repo_path) / path).resolve()) - logger.info(f" ├─ Path #{i}: {path}") - logger.info(f" │ └─ Normalized: {normalized}") - -logger.info(f" └─ Normalized {len(additional_paths_normalized)} additional path(s)") -``` - -**What It Shows:** -- Receipt of additional paths from CLI -- Original relative path from config -- Normalized absolute path -- Path count before and after normalization - ---- - -### 5. Graph Builder Layer (`codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py`) - -#### `build_dependency_graph()` Method (Lines 44-52) -**Added Logging:** -```python -logger.info(f"🔍 Multi-path mode enabled: analyzing {len(repo_paths)} source paths") -logger.info(f" ├─ Primary: {repo_paths[0]}") -for i, path in enumerate(repo_paths[1:], 1): - logger.info(f" └─ Additional #{i}: {path}") -# OR (single-path mode) -logger.info(f"📁 Single-path mode: analyzing {repo_paths[0]}") -``` - -**What It Shows:** -- Mode detection (multi-path vs single-path) -- All source paths being analyzed -- Primary vs additional path distinction - ---- - -## Complete Logging Flow - -### Startup Sequence -``` -1. CLI receives additional_paths from config - └─ Logs raw paths and normalized paths - -2. BackendConfig.from_cli() creates config - └─ Sets additional_source_paths - -3. Config.validate_source_paths() validates all paths - └─ Logs validation for primary + additional paths - -4. Config.is_multi_path_mode() checks mode - └─ Logs mode detection - -5. Config.all_source_paths property accessed - └─ Logs all paths being analyzed -``` - -### Analysis Sequence -``` -6. DependencyGraphBuilder.build_dependency_graph() - └─ Logs mode and all source paths - -7. RepoAnalyzer._analyze_multiple_repositories() - └─ Logs file tree building per path - └─ Logs file counts and sizes per namespace - └─ Logs merged statistics - -8. DependencyParser._parse_multiple_repositories() - └─ Logs multi-path detection - └─ Logs namespace generation per path - └─ Logs component counts per namespace - └─ Logs component types per namespace - -9. DependencyParser._resolve_cross_namespace_dependencies() - └─ Logs cross-namespace dependencies found - └─ Logs resolution summary -``` - ---- - -## Log Level Breakdown - -### INFO Level (Always Visible) -- Multi-path mode detection -- Path lists and counts -- Namespace assignments -- Component counts per namespace -- Component type breakdowns -- Cross-namespace dependency summaries -- Final statistics - -### DEBUG Level (Verbose Mode Only) -- Property access logs -- Detailed validation steps -- Individual cross-namespace dependencies -- Internal state transitions - -### WARNING Level (Errors/Issues) -- No files found in path -- Leaf nodes not found in components -- Invalid paths or permissions - ---- - -## Testing Recommendations - -### Test Multi-Path Mode -```bash -codewiki generate \ - --repo-path=/path/to/primary \ - --output-dir=output \ - --additional-paths=../ui-kit,../shared-lib \ - --verbose -``` - -**Expected Log Output:** -``` -🔍 CLI received 2 additional path(s): - ├─ Path #1: ../ui-kit - │ └─ Normalized: /absolute/path/to/ui-kit - ├─ Path #2: ../shared-lib - │ └─ Normalized: /absolute/path/to/shared-lib - └─ Normalized 2 additional path(s) - -🔍 Validating source paths... - ├─ ✓ Primary path valid: /path/to/primary - ├─ Validating 2 additional path(s)... - │ ├─ ✓ Additional path #1 valid: /absolute/path/to/ui-kit - │ ├─ ✓ Additional path #2 valid: /absolute/path/to/shared-lib - └─ ✓ All 2 additional path(s) validated - -🔍 Multi-path mode enabled: analyzing 3 source paths - ├─ Primary: /path/to/primary - ├─ Additional #1: /absolute/path/to/ui-kit - └─ Additional #2: /absolute/path/to/shared-lib - -🔍 Analyzing 3 repository paths... - ├─ [1/3] Building file tree for: /path/to/primary - │ └─ Namespace: 'primary' - │ └─ Found 150 files (2500.00 KB) - ├─ [2/3] Building file tree for: /absolute/path/to/ui-kit - │ └─ Namespace: 'ui-kit' - │ └─ Found 75 files (1200.00 KB) - ├─ [3/3] Building file tree for: /absolute/path/to/shared-lib - │ └─ Namespace: 'shared-lib' - │ └─ Found 50 files (800.00 KB) - └─ ✓ Merged 3 repositories: - ├─ Total files: 275 - ├─ Total size: 4500.00 KB - └─ Namespaces: primary, ui-kit, shared-lib - -🔍 Multi-path mode detected: analyzing 3 source directories - ├─ Primary: /path/to/primary - ├─ Additional #1: /absolute/path/to/ui-kit - └─ Additional #2: /absolute/path/to/shared-lib - -📂 Analyzing path 1/3: /path/to/primary - └─ Namespace: 'primary' -📊 Namespace 'primary': found 200 components - └─ Component types: - • function: 120 - • class: 50 - • method: 30 - -📂 Analyzing path 2/3: /absolute/path/to/ui-kit - └─ Namespace: 'ui-kit' -📊 Namespace 'ui-kit': found 100 components - └─ Component types: - • function: 60 - • class: 40 - -📂 Analyzing path 3/3: /absolute/path/to/shared-lib - └─ Namespace: 'shared-lib' -📊 Namespace 'shared-lib': found 50 components - └─ Component types: - • function: 30 - • class: 20 - -🔗 Resolving cross-namespace dependencies... - └─ ✓ Resolved 15 cross-namespace dependencies - -📊 Multi-path analysis complete: - ├─ Total components: 350 - ├─ Total modules: 12 - └─ Namespaces: 3 -``` - -### Test Single-Path Mode (Backward Compatibility) -```bash -codewiki generate \ - --repo-path=/path/to/single \ - --output-dir=output \ - --verbose -``` - -**Expected Log Output:** -``` -🔍 all_source_paths property accessed (single-path mode): /path/to/single - -🔍 Validating source paths... - ├─ ✓ Primary path valid: /path/to/single - └─ No additional paths to validate (single-path mode) - -🔍 Multi-path mode: DISABLED (single-path) - -📁 Single-path mode: analyzing /path/to/single - -🔍 Parsing repository files... - └─ Parsed 200 components total - └─ Component types found: - • function: 120 - • class: 50 - • method: 30 -``` - ---- - -## Verification Checklist - -- [x] Config layer logs path access and validation -- [x] Config layer logs mode detection -- [x] CLI layer logs path normalization -- [x] Graph builder logs source paths -- [x] Analyzer layer logs file tree merging -- [x] Parser layer logs namespace generation -- [x] Parser layer logs component counts per namespace -- [x] Parser layer logs component types per namespace -- [x] Parser layer logs cross-namespace dependencies -- [x] All syntax validated with py_compile -- [x] Backward compatibility maintained (single-path mode) - ---- - -## Benefits of This Logging - -1. **Debugging**: Easily identify which path is causing issues -2. **Progress Tracking**: See real-time progress for each repository -3. **Verification**: Confirm all paths are being analyzed -4. **Performance**: Monitor file counts and sizes per path -5. **Dependency Analysis**: Track cross-namespace dependencies -6. **Transparency**: Complete visibility into multi-path pipeline - ---- - -## Log Format Consistency - -All logs use consistent emoji prefixes: -- 🔍 = Detection/Analysis -- 📂 = File/Directory operations -- 📊 = Statistics/Summaries -- 🔗 = Dependencies -- ✓ = Success -- ⚠️ = Warning - -Tree structure formatting: -- `├─` = Branch -- `└─` = Last branch -- `│` = Continuation -- `•` = List item diff --git a/MULTI_PATH_QUICK_REFERENCE.md b/MULTI_PATH_QUICK_REFERENCE.md deleted file mode 100644 index c27b7884..00000000 --- a/MULTI_PATH_QUICK_REFERENCE.md +++ /dev/null @@ -1,291 +0,0 @@ -# Multi-Path Support - Quick Reference - -## Basic Usage - -### Single Path (Backward Compatible) -```python -from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser - -parser = DependencyParser("/path/to/repo") -components = parser.parse_repository() -# Component IDs: "src.module.Component" -``` - -### Multiple Paths -```python -parser = DependencyParser([ - "/path/to/openframe-frontend", - "/path/to/ui-kit" -]) -components = parser.parse_repository() -# Component IDs: "openframe-frontend.src.module.Component" -# "ui-kit.components.Button" -``` - -## Component ID Format - -| Mode | Format | Example | -|------|--------|---------| -| Single | `{module}.{component}` | `src.components.Button` | -| Multi | `{namespace}.{module}.{component}` | `ui-kit.src.components.Button` | - -## Key Methods - -### DependencyParser - -```python -parser = DependencyParser( - repo_path: Union[str, List[str]], # Single or multiple paths - include_patterns: List[str] = None, # ["*.py", "*.ts"] - exclude_patterns: List[str] = None # ["*test*"] -) - -# Parse repositories -components: Dict[str, Node] = parser.parse_repository() - -# Save to JSON -parser.save_dependency_graph("/path/to/output.json") - -# Access namespace info (multi-path only) -namespace = parser._get_namespace_from_path("/path/to/repo") -``` - -### RepoAnalyzer - -```python -analyzer = RepoAnalyzer( - include_patterns: List[str] = None, - exclude_patterns: List[str] = None -) - -# Single path -result = analyzer.analyze_repository_structure("/path/to/repo") - -# Multiple paths -result = analyzer.analyze_repository_structure([ - "/path/to/repo1", - "/path/to/repo2" -]) - -# Result structure: -{ - "file_tree": {...}, - "summary": { - "total_files": int, - "total_size_kb": float, - "repositories": int, # Multi-path only - "namespaces": List[str] # Multi-path only - } -} -``` - -## Component Object (Node) - -```python -component = components["namespace.module.Component"] - -# Core attributes -component.id # Namespaced ID -component.name # Original component name -component.component_type # "function", "class", "method" -component.file_path # Absolute path -component.relative_path # Relative to repo -component.depends_on # Set of dependency IDs (namespaced) - -# Code details -component.source_code # Full source code -component.start_line # Starting line number -component.end_line # Ending line number -component.docstring # Component docstring -component.parameters # List of parameters - -# Additional info -component.node_type # AST node type -component.class_name # Parent class (if method) -component.base_classes # Inheritance (if class) -``` - -## Dependencies - -### Intra-Namespace (Same Repository) -```python -{ - "id": "openframe-frontend.src.App.main", - "depends_on": [ - "openframe-frontend.src.utils.logger", - "openframe-frontend.src.config.settings" - ] -} -``` - -### Cross-Namespace (Different Repositories) -```python -{ - "id": "openframe-frontend.src.App.main", - "depends_on": [ - "openframe-frontend.src.utils.logger", # Same namespace - "ui-kit.components.Button", # Cross-namespace - "shared-lib.api.fetchData" # Cross-namespace - ] -} -``` - -## Common Patterns - -### Filtering Components by Namespace -```python -parser = DependencyParser(["/repo1", "/repo2"]) -components = parser.parse_repository() - -# Get components from specific namespace -repo1_components = { - comp_id: comp for comp_id, comp in components.items() - if comp_id.startswith("repo1.") -} -``` - -### Finding Cross-Namespace Dependencies -```python -for comp_id, component in components.items(): - comp_namespace = comp_id.split('.')[0] - cross_deps = [ - dep for dep in component.depends_on - if dep.split('.')[0] != comp_namespace - ] - if cross_deps: - print(f"{comp_id} has {len(cross_deps)} cross-namespace deps") -``` - -### Grouping by Namespace -```python -from collections import defaultdict - -by_namespace = defaultdict(list) -for comp_id in components.keys(): - namespace = comp_id.split('.')[0] - by_namespace[namespace].append(comp_id) - -for namespace, comp_ids in by_namespace.items(): - print(f"{namespace}: {len(comp_ids)} components") -``` - -## File Patterns - -### Include Patterns -```python -# Python only -include_patterns = ["*.py"] - -# TypeScript/JavaScript -include_patterns = ["*.ts", "*.tsx", "*.js", "*.jsx"] - -# Multiple languages -include_patterns = ["*.py", "*.ts", "*.tsx", "*.cs", "*.java"] -``` - -### Exclude Patterns -```python -exclude_patterns = [ - "*test*", # Test files - "*spec*", # Spec files - "node_modules/*", # Dependencies - "build/*", # Build output - "dist/*", # Distribution files - ".git/*", # Git directory - "__pycache__/*" # Python cache -] -``` - -## Error Handling - -```python -try: - parser = DependencyParser(["/repo1", "/repo2"]) - components = parser.parse_repository() -except FileNotFoundError as e: - print(f"Repository not found: {e}") -except PermissionError as e: - print(f"Permission denied: {e}") -except Exception as e: - print(f"Parsing error: {e}") -``` - -## Performance Tips - -1. **Use Specific Include Patterns**: Only analyze relevant file types -2. **Aggressive Exclude Patterns**: Skip test files, dependencies, build outputs -3. **Limit Number of Paths**: More paths = longer analysis time -4. **Filter Early**: Use `filtered_folders` parameter when possible - -## Output Format (JSON) - -```json -{ - "namespace.module.Component": { - "id": "namespace.module.Component", - "name": "Component", - "component_type": "class", - "file_path": "/absolute/path/to/file.py", - "relative_path": "src/module/file.py", - "source_code": "class Component:\n ...", - "start_line": 10, - "end_line": 50, - "depends_on": [ - "namespace.module.Dependency1", - "other-namespace.utils.Helper" - ], - "docstring": "Component docstring", - "parameters": [], - "node_type": "class", - "has_docstring": true - } -} -``` - -## Backward Compatibility - -All existing single-path code works without changes: - -```python -# ✅ Still works -parser = DependencyParser("/path/to/repo") -parser.repo_path # Still available -components = parser.parse_repository() # Same behavior -``` - -## Migration Checklist - -- [ ] Update `DependencyParser` constructor calls to use list of paths -- [ ] Update code that assumes non-namespaced component IDs -- [ ] Update dependency resolution logic for cross-namespace refs -- [ ] Update JSON output parsing if used downstream -- [ ] Test with actual multi-path repositories -- [ ] Update documentation/examples - -## Common Issues - -### Issue: "Component not found" -**Cause**: Cross-namespace dependency resolution failed -**Solution**: Check component names match exactly, verify both repos were parsed - -### Issue: "Duplicate component IDs" -**Cause**: Two repos with same directory name -**Solution**: Rename one repository directory or use unique paths - -### Issue: "Performance degradation" -**Cause**: Too many files being analyzed -**Solution**: Add more exclude patterns, use specific include patterns - -### Issue: "Missing dependencies" -**Cause**: Component exists but not in dependency graph -**Solution**: Check include/exclude patterns, verify file was parsed - -## Examples Location - -See `/examples/multi_path_example.py` for complete working examples. - -## Documentation - -- Full Implementation: `MULTI_PATH_IMPLEMENTATION.md` -- API Reference: Component docstrings in source files -- Examples: `examples/multi_path_example.py` diff --git a/SECURITY_ANALYSIS_ADDITIONAL_PATHS.md b/SECURITY_ANALYSIS_ADDITIONAL_PATHS.md deleted file mode 100644 index dd530058..00000000 --- a/SECURITY_ANALYSIS_ADDITIONAL_PATHS.md +++ /dev/null @@ -1,852 +0,0 @@ -# Security & Edge Case Analysis: Additional Paths Implementation - -**Analysis Date:** 2026-02-02 -**Analyzer:** Claude Sonnet 4.5 -**CodeWiki Version:** Latest (main branch) -**Feature:** `--additional-paths` CLI parameter and `additional_source_paths` config field - ---- - -## Executive Summary - -✅ **SECURITY STATUS: SAFE** - -The additional paths implementation is **secure and well-designed**. After comprehensive analysis of all code paths from CLI input through backend validation, NO security vulnerabilities were found. The implementation follows security best practices including: - -- Absolute path normalization prevents directory traversal -- Existence validation prevents symlink attacks -- Read permission checks prevent privilege escalation -- No shell command execution (pure Python path operations) -- Proper error handling with descriptive messages - -**Recommended Actions:** -1. ✅ Safe to use in production -2. 📝 Add integration tests for edge cases (see Test Plan section) -3. 🔒 Consider adding optional path whitelist for enterprise deployments - ---- - -## 1. Security Analysis - -### 1.1 Path Traversal Attacks ✅ SAFE - -**Attack Vector:** `../../etc/passwd` or similar relative paths to escape repository root - -**Code Analysis:** - -```python -# Location: codewiki/cli/commands/generate.py:392-422 -additional_paths_list = parse_patterns(additional_paths) - -# Validate that paths exist -for path in additional_paths_list: - full_path = repo_path / path # Joins with repo_path - if not full_path.exists(): - invalid_paths.append(path) -``` - -```python -# Location: codewiki/cli/adapters/doc_generator.py:150 -normalized = str((Path(self.repo_path) / path).resolve()) -``` - -**Protection Mechanisms:** -1. **Path.resolve()**: Automatically canonicalizes paths, resolving `.` and `..` -2. **Path joining**: Uses `/` operator which normalizes paths -3. **Existence validation**: Path must exist before processing - -**Test Results:** - -| Attack Input | Normalized Output | Result | -|--------------|------------------|--------| -| `../../etc/passwd` | `/etc/passwd` (if exists) | ✅ Blocked by validation if outside repo | -| `../../../` | Parent dirs resolved | ✅ Validated against existence | -| `vendor/../../../etc` | `/etc` | ✅ Must exist under repo_path | - -**Verdict:** ✅ **PROTECTED** - Path traversal automatically resolved and validated - ---- - -### 1.2 Symlink Attacks ✅ SAFE - -**Attack Vector:** Symlink to sensitive system files or directories - -**Code Analysis:** - -```python -# Location: codewiki/src/config.py:180-188 -if not os.path.exists(path): - raise ValueError(f"Additional source path #{i} does not exist: {path}") -if not os.path.isdir(path): - raise ValueError(f"Additional source path #{i} is not a directory: {path}") - -# Check for read access -if not os.access(path, os.R_OK): - raise OSError(f"Additional source path #{i} is not readable: {path}") -``` - -**Protection Mechanisms:** -1. **os.path.exists()**: Follows symlinks by default (checks target exists) -2. **os.path.isdir()**: Validates target is a directory (not a file) -3. **os.access(R_OK)**: Verifies read permissions on target - -**Attack Scenarios:** - -| Symlink Target | Validation Behavior | Result | -|----------------|---------------------|--------| -| `ln -s /etc/passwd secrets` | ❌ isdir() fails (file, not dir) | ✅ Blocked | -| `ln -s /root/.ssh config` | ❌ access(R_OK) fails (no perms) | ✅ Blocked | -| `ln -s /var/www/html external` | ✅ Passes if readable dir | ⚠️ User responsibility | - -**Verdict:** ✅ **PROTECTED** - Symlinks must point to readable directories. User is responsible for what they symlink. - ---- - -### 1.3 Command Injection ✅ SAFE - -**Attack Vector:** Shell metacharacters in paths like `; rm -rf /` or `$(whoami)` - -**Code Analysis:** - -```python -# ALL path operations use Python pathlib/os modules - NO shell execution -Path(self.repo_path) / path # Pure Python path joining -os.path.exists(path) # Pure syscall, no shell -os.path.isdir(path) # Pure syscall, no shell -os.access(path, os.R_OK) # Pure syscall, no shell -``` - -**Protection Mechanisms:** -1. **No subprocess calls**: Paths never passed to shell -2. **pathlib abstraction**: Safe object-oriented path handling -3. **os module syscalls**: Direct system calls, not shell commands - -**Test Results:** - -| Injection Attempt | Processing | Result | -|-------------------|------------|--------| -| `vendor; rm -rf /` | Treated as literal path | ✅ File not found error | -| `$(whoami)` | Literal filename | ✅ File not found error | -| `id` | Literal filename | ✅ File not found error | -| `external && malicious` | Literal path | ✅ File not found error | - -**Verdict:** ✅ **COMPLETELY SAFE** - No shell execution anywhere in code path - ---- - -### 1.4 Privilege Escalation ✅ SAFE - -**Attack Vector:** Access files/directories outside user's permissions - -**Code Analysis:** - -```python -# Location: codewiki/src/config.py:187-188 -if not os.access(path, os.R_OK): - raise OSError(f"Additional source path #{i} is not readable: {path}") -``` - -**Protection Mechanisms:** -1. **os.access(R_OK)**: Checks effective UID/GID permissions -2. **No SUID operations**: CodeWiki runs with user's permissions -3. **Fail-fast validation**: Errors before any processing begins - -**Verdict:** ✅ **SAFE** - Cannot escalate beyond user's existing permissions - ---- - -## 2. Edge Case Analysis - -### 2.1 Empty Environment Variable ✅ HANDLED - -**Test:** -```bash -CODEWIKI_ADDITIONAL_PATHS="" codewiki generate -``` - -**Behavior:** -```python -# codewiki/cli/commands/generate.py:142-145 -@click.option( - "--additional-paths", - type=str, - default=None, # ✅ Defaults to None, not empty string -``` - -**Verdict:** ✅ Treated as unset, no additional paths loaded - ---- - -### 2.2 Paths with Spaces ✅ HANDLED - -**Test:** -```bash -codewiki generate --additional-paths "vendor packages,external deps" -``` - -**Behavior:** -```python -# codewiki/cli/commands/generate.py:35-39 -def parse_patterns(patterns_str: str) -> List[str]: - if not patterns_str: - return [] - return [p.strip() for p in patterns_str.split(',') if p.strip()] - # ✅ Result: ["vendor packages", "external deps"] -``` - -**Verdict:** ✅ Spaces in paths work correctly (after strip()) - ---- - -### 2.3 Special Characters ✅ HANDLED - -**Test:** -```bash -codewiki generate --additional-paths "vendor-2.0,external_libs,deps@v3" -``` - -**Behavior:** -- Hyphens: ✅ Valid in filesystem paths -- Underscores: ✅ Valid in filesystem paths -- `@` symbol: ✅ Valid in filesystem paths - -**Verdict:** ✅ All common special chars work - ---- - -### 2.4 Non-Existent Directories ✅ HANDLED - -**Test:** -```bash -codewiki generate --additional-paths "vendor,nonexistent" -``` - -**Behavior:** -```python -# codewiki/cli/commands/generate.py:400-418 -invalid_paths = [] -for path in additional_paths_list: - full_path = repo_path / path - if not full_path.exists(): - invalid_paths.append(path) - -if invalid_paths: - raise RepositoryError( - f"Additional paths not found:\n" - f" {', '.join(invalid_paths)}\n\n" - f"Please ensure all paths exist relative to repository root:\n" - f" Repository: {repo_path}\n" - f" Invalid paths: {invalid_paths}" - ) -``` - -**Output:** -``` -Error: Additional paths not found: - nonexistent - -Please ensure all paths exist relative to repository root: - Repository: /Users/user/myrepo - Invalid paths: ['nonexistent'] -``` - -**Verdict:** ✅ Clear error message with helpful context - ---- - -### 2.5 Circular Symlinks ✅ HANDLED - -**Test:** -```bash -ln -s link1 link2 -ln -s link2 link1 -codewiki generate --additional-paths "link1" -``` - -**Behavior:** -```python -# Python's os.path.exists() handles circular symlinks -# Returns False for unresolvable symlinks -if not os.path.exists(path): # ✅ Returns False - raise ValueError(f"Additional source path does not exist: {path}") -``` - -**Verdict:** ✅ Detected as non-existent path - ---- - -### 2.6 Very Long Paths (> 4096 chars) ⚠️ SYSTEM LIMIT - -**Test:** -```bash -# Create deeply nested directory -long_path = "a/" * 2000 # 4000 characters -codewiki generate --additional-paths "$long_path" -``` - -**Behavior:** -- Linux PATH_MAX: 4096 bytes (including null terminator) -- macOS PATH_MAX: 1024 bytes -- Windows MAX_PATH: 260 characters (legacy) / 32,767 (Unicode) - -**Python Handling:** -```python -# Python pathlib raises OSError if path too long -try: - Path(long_path).resolve() -except OSError as e: - # "File name too long" error -``` - -**Verdict:** ⚠️ **SYSTEM LIMITATION** - Handled by OS, clear error message - -**Recommendation:** Add documentation note about PATH_MAX limits - ---- - -### 2.7 Multiple Comma-Separated Paths ✅ HANDLED - -**Test:** -```bash -codewiki generate --additional-paths "vendor,external,deps,third_party" -``` - -**Behavior:** -```python -# codewiki/cli/commands/generate.py:392 -additional_paths_list = parse_patterns(additional_paths) -# Result: ["vendor", "external", "deps", "third_party"] - -# All validated in loop: -for i, path in enumerate(additional_paths_list, 1): - logger.debug(f"│ ├─ Path #{i}: {path}") -``` - -**Verdict:** ✅ Fully supported, all paths validated - ---- - -### 2.8 Relative vs Absolute Paths ✅ NORMALIZED - -**Test:** -```bash -# Relative paths -codewiki generate --additional-paths "vendor,./external,../sibling" - -# Absolute paths -codewiki generate --additional-paths "/opt/libs,/usr/local/deps" -``` - -**Behavior:** -```python -# codewiki/cli/adapters/doc_generator.py:150 -normalized = str((Path(self.repo_path) / path).resolve()) - -# Relative: joined with repo_path, then resolved -# Absolute: resolve() keeps absolute if valid -``` - -**Results:** - -| Input | Repo Path | Normalized Output | -|-------|-----------|------------------| -| `vendor` | `/home/user/repo` | `/home/user/repo/vendor` | -| `./external` | `/home/user/repo` | `/home/user/repo/external` | -| `../sibling` | `/home/user/repo` | `/home/user/sibling` | -| `/opt/libs` | `/home/user/repo` | `/opt/libs` | - -**Verdict:** ✅ Both relative and absolute paths work correctly - ---- - -### 2.9 Paths with Trailing Slash ✅ NORMALIZED - -**Test:** -```bash -codewiki generate --additional-paths "vendor/,external" -``` - -**Behavior:** -```python -# pathlib normalizes trailing slashes automatically -Path("vendor/").resolve() # → /absolute/path/vendor (no trailing slash) -``` - -**Verdict:** ✅ Trailing slashes automatically removed - ---- - -## 3. Error Handling Analysis - -### 3.1 Path Validation Failure ✅ EXCELLENT - -**Location:** `codewiki/cli/commands/generate.py:408-418` - -```python -if invalid_paths: - raise RepositoryError( - f"Additional paths not found:\n" - f" {', '.join(invalid_paths)}\n\n" - f"Please ensure all paths exist relative to repository root:\n" - f" Repository: {repo_path}\n" - f" Invalid paths: {invalid_paths}" - ) -``` - -**Quality:** -- ✅ Clear error type (`RepositoryError`) -- ✅ Lists all invalid paths -- ✅ Shows repository context -- ✅ Suggests solution ("ensure paths exist") - ---- - -### 3.2 Permission Errors ✅ EXCELLENT - -**Location:** `codewiki/src/config.py:187-188` - -```python -if not os.access(path, os.R_OK): - raise OSError(f"Additional source path #{i} is not readable: {path}") -``` - -**Quality:** -- ✅ Clear error type (`OSError`) -- ✅ Identifies which path failed (by number and path) -- ✅ Specific issue ("not readable") - ---- - -### 3.3 Type Errors ✅ HANDLED - -**Scenario:** Non-string passed to parse_patterns - -```python -def parse_patterns(patterns_str: str) -> List[str]: - if not patterns_str: # ✅ Handles None - return [] - return [p.strip() for p in patterns_str.split(',') if p.strip()] -``` - -**Test:** -- `None` → `[]` (empty list) -- Empty string → `[]` (empty list) -- Integer → Would fail at `.split()` with AttributeError (handled by Click type validation) - -**Verdict:** ✅ Type hints + Click validation prevent bad input - ---- - -### 3.4 Graceful Exit on Validation Failure ✅ PERFECT - -**Code Flow:** -```python -# 1. Early validation in CLI command -if invalid_paths: - raise RepositoryError(...) # Exit code based on error type - -# 2. Exception handling -except RepositoryError as e: - logger.error(e.message) - sys.exit(e.exit_code) # ✅ Clean exit with proper code -``` - -**Verdict:** ✅ No partial state changes, clean exit - ---- - -### 3.5 Logging Coverage ✅ EXCELLENT - -**Verbose Mode Output:** -```python -# codewiki/cli/commands/generate.py:389-421 -if verbose: - logger.debug("🔍 Stage 1.3: Additional Paths Validation") - logger.debug(f"├─ Parsed {len(additional_paths_list)} additional path(s)") - for i, path in enumerate(additional_paths_list, 1): - logger.debug(f"│ ├─ Path #{i}: {path}") - # ... validation ... - logger.debug(f"│ └─ ✓ {path} (exists)") - logger.debug(f"└─ All additional paths validated") -``` - -**Verdict:** ✅ Comprehensive logging for debugging - ---- - -## 4. Test Plan - -### 4.1 Unit Tests (Recommended) - -**File:** `tests/test_additional_paths.py` - -```python -import pytest -from pathlib import Path -from codewiki.cli.commands.generate import parse_patterns -from codewiki.src.config import Config - -class TestParsePatterns: - def test_empty_string(self): - assert parse_patterns("") == [] - - def test_single_path(self): - assert parse_patterns("vendor") == ["vendor"] - - def test_multiple_paths(self): - assert parse_patterns("vendor,external,deps") == ["vendor", "external", "deps"] - - def test_spaces_stripped(self): - assert parse_patterns("vendor , external , deps") == ["vendor", "external", "deps"] - - def test_trailing_commas(self): - assert parse_patterns("vendor,external,") == ["vendor", "external"] - -class TestConfigValidation: - def test_nonexistent_path(self, tmp_path): - config = Config.from_cli( - repo_path=str(tmp_path), - output_dir=str(tmp_path / "output"), - main_model="test", - cluster_model="test", - fallback_model="test", - cluster_api_key="test", - main_api_key="test", - fallback_api_key="test", - cluster_base_url="http://test", - main_base_url="http://test", - fallback_base_url="http://test", - additional_source_paths=[str(tmp_path / "nonexistent")] - ) - - with pytest.raises(ValueError, match="does not exist"): - config.validate_source_paths() - - def test_valid_additional_paths(self, tmp_path): - # Create test directories - vendor = tmp_path / "vendor" - vendor.mkdir() - external = tmp_path / "external" - external.mkdir() - - config = Config.from_cli( - repo_path=str(tmp_path), - output_dir=str(tmp_path / "output"), - main_model="test", - cluster_model="test", - fallback_model="test", - cluster_api_key="test", - main_api_key="test", - fallback_api_key="test", - cluster_base_url="http://test", - main_base_url="http://test", - fallback_base_url="http://test", - additional_source_paths=[str(vendor), str(external)] - ) - - # Should not raise - config.validate_source_paths() - assert config.is_multi_path_mode() is True - assert len(config.all_source_paths) == 3 # repo + 2 additional -``` - ---- - -### 4.2 Integration Tests (Recommended) - -**File:** `tests/integration/test_cli_additional_paths.py` - -```python -import subprocess -import pytest -from pathlib import Path - -def test_cli_nonexistent_path(tmp_path): - """Test CLI with nonexistent path""" - result = subprocess.run( - ["codewiki", "generate", "--additional-paths", "nonexistent"], - cwd=str(tmp_path), - capture_output=True, - text=True - ) - - assert result.returncode != 0 - assert "Additional paths not found" in result.stderr - assert "nonexistent" in result.stderr - -def test_cli_special_characters(tmp_path): - """Test paths with special characters""" - special_dir = tmp_path / "vendor-2.0" - special_dir.mkdir() - - result = subprocess.run( - ["codewiki", "generate", "--additional-paths", "vendor-2.0", "--verbose"], - cwd=str(tmp_path), - capture_output=True, - text=True - ) - - assert "vendor-2.0" in result.stdout or result.returncode == 0 - -def test_cli_symlink_to_directory(tmp_path): - """Test symlink to valid directory""" - real_dir = tmp_path / "real" - real_dir.mkdir() - - link = tmp_path / "link" - link.symlink_to(real_dir) - - result = subprocess.run( - ["codewiki", "generate", "--additional-paths", "link", "--verbose"], - cwd=str(tmp_path), - capture_output=True, - text=True - ) - - # Should work (symlink to dir) - assert "All additional paths validated" in result.stdout - -def test_cli_symlink_to_file(tmp_path): - """Test symlink to file (should fail)""" - real_file = tmp_path / "file.txt" - real_file.write_text("test") - - link = tmp_path / "link" - link.symlink_to(real_file) - - result = subprocess.run( - ["codewiki", "generate", "--additional-paths", "link"], - cwd=str(tmp_path), - capture_output=True, - text=True - ) - - assert result.returncode != 0 - assert "not a directory" in result.stderr.lower() -``` - ---- - -### 4.3 Security Tests (Critical) - -**File:** `tests/security/test_path_traversal.py` - -```python -import pytest -from pathlib import Path -from codewiki.src.config import Config - -class TestPathTraversalProtection: - def test_relative_parent_traversal(self, tmp_path): - """Test ../../etc/passwd style attacks""" - config = Config.from_cli( - repo_path=str(tmp_path), - output_dir=str(tmp_path / "output"), - main_model="test", - cluster_model="test", - fallback_model="test", - cluster_api_key="test", - main_api_key="test", - fallback_api_key="test", - cluster_base_url="http://test", - main_base_url="http://test", - fallback_base_url="http://test", - additional_source_paths=["../../etc/passwd"] - ) - - with pytest.raises((ValueError, FileNotFoundError)): - config.validate_source_paths() - - def test_absolute_system_path(self, tmp_path): - """Test absolute path to system directory""" - config = Config.from_cli( - repo_path=str(tmp_path), - output_dir=str(tmp_path / "output"), - main_model="test", - cluster_model="test", - fallback_model="test", - cluster_api_key="test", - main_api_key="test", - fallback_api_key="test", - cluster_base_url="http://test", - main_base_url="http://test", - fallback_base_url="http://test", - additional_source_paths=["/etc"] - ) - - # This would work if /etc is readable by user - # The question is: should we allow this? - # Current implementation: YES (user is responsible) -``` - ---- - -## 5. Bugs Found - -### 🐛 None Found - -After comprehensive analysis, **no bugs were identified** in the additional paths implementation. - ---- - -## 6. Security Hardening Recommendations - -### 6.1 Optional Path Whitelist (LOW PRIORITY) - -**For enterprise deployments**, consider adding a configuration option to restrict additional paths to specific directories: - -```python -# config.py -class Config: - allowed_additional_paths_prefix: Optional[List[str]] = None - - def validate_source_paths(self) -> None: - if self.allowed_additional_paths_prefix: - for path in self.additional_source_paths: - if not any(path.startswith(prefix) for prefix in self.allowed_additional_paths_prefix): - raise ValueError( - f"Additional path {path} not allowed. " - f"Must start with one of: {self.allowed_additional_paths_prefix}" - ) -``` - -**Use Case:** Prevent users from accidentally including system directories - -**Priority:** LOW (current implementation is already safe) - ---- - -### 6.2 Document PATH_MAX Limits (DOCUMENTATION) - -Add to documentation: - -```markdown -## Path Length Limits - -Additional paths are subject to operating system PATH_MAX limits: -- **Linux:** 4,096 characters -- **macOS:** 1,024 characters -- **Windows:** 260 characters (legacy) or 32,767 (long path mode) - -If you encounter "File name too long" errors, reduce your directory nesting depth. -``` - ---- - -### 6.3 Add Warning for System Directories (UX IMPROVEMENT) - -Consider warning users if they add paths outside their home directory: - -```python -def validate_source_paths(self) -> None: - # ... existing validation ... - - # Warn for system paths - home_dir = Path.home() - for path in self.additional_source_paths: - abs_path = Path(path).resolve() - if not abs_path.is_relative_to(home_dir): - logger.warning( - f"⚠️ Additional path is outside your home directory: {path}\n" - f" Ensure you have proper permissions and this is intentional." - ) -``` - ---- - -## 7. Conclusion - -### ✅ Security: EXCELLENT - -The additional paths implementation demonstrates **strong security practices**: - -1. ✅ No command injection vulnerabilities (pure Python path operations) -2. ✅ Path traversal automatically resolved and validated -3. ✅ Symlink attacks mitigated by type and permission checks -4. ✅ Cannot escalate beyond user's existing permissions -5. ✅ Comprehensive validation with clear error messages - -### ✅ Edge Case Handling: ROBUST - -All edge cases are properly handled: - -1. ✅ Empty/missing values → default to None -2. ✅ Special characters → work correctly -3. ✅ Non-existent paths → clear error message -4. ✅ Circular symlinks → detected as non-existent -5. ✅ Multiple paths → all validated -6. ✅ Relative/absolute → both normalized correctly - -### ✅ Error Handling: EXCELLENT - -Error handling is comprehensive and user-friendly: - -1. ✅ Descriptive error messages with context -2. ✅ Proper exception types -3. ✅ Clean exit codes -4. ✅ Detailed logging in verbose mode -5. ✅ No partial state on failure - -### 📊 Overall Grade: A+ - -The additional paths feature is **production-ready** with excellent security and robustness. No critical issues found. - ---- - -## Appendix A: Code Flow Diagram - -``` -User Input: --additional-paths "vendor,external" - ↓ -CLI Command (generate.py:142-166) - ├─ Parse comma-separated string → ["vendor", "external"] - ├─ Validate paths exist (repo_path / path) - ├─ Raise RepositoryError if any missing - └─ Pass to doc_generator - ↓ -Doc Generator (doc_generator.py:140-155) - ├─ Normalize to absolute paths with resolve() - ├─ Log normalized paths in verbose mode - └─ Pass to BackendConfig.from_cli() - ↓ -Backend Config (config.py:326-576) - ├─ Store in additional_source_paths field - ├─ Call validate_source_paths() - │ ├─ Check os.path.exists() - │ ├─ Check os.path.isdir() - │ └─ Check os.access(R_OK) - └─ Use in all_source_paths property - ↓ -Documentation Generation - ├─ Iterate over all_source_paths - ├─ Analyze dependencies from each path - └─ Merge into unified documentation -``` - ---- - -## Appendix B: Test Coverage Summary - -| Test Category | Status | Priority | -|---------------|--------|----------| -| Path Traversal Security | ✅ Verified Safe | CRITICAL | -| Symlink Security | ✅ Verified Safe | CRITICAL | -| Command Injection | ✅ Verified Safe | CRITICAL | -| Privilege Escalation | ✅ Verified Safe | CRITICAL | -| Edge Cases | ✅ All Handled | HIGH | -| Error Handling | ✅ Excellent | HIGH | -| Unit Tests | 📝 Recommended | MEDIUM | -| Integration Tests | 📝 Recommended | MEDIUM | -| Security Tests | 📝 Recommended | HIGH | - -**Recommended Next Steps:** -1. Implement unit tests from Section 4.1 -2. Add integration tests from Section 4.2 -3. Run security tests from Section 4.3 -4. Add documentation note about PATH_MAX limits - ---- - -**Report Prepared By:** Claude Sonnet 4.5 -**Verification Status:** ✅ All code paths analyzed -**Confidence Level:** 99% (manual testing recommended for 100%) diff --git a/codewiki/cli/adapters/.doc_generator.md b/codewiki/cli/adapters/.doc_generator.md index 15a95850..16473fa0 100644 --- a/codewiki/cli/adapters/.doc_generator.md +++ b/codewiki/cli/adapters/.doc_generator.md @@ -8,7 +8,7 @@ CLI adapter that wraps the backend `DocumentationGenerator` to add CLI-specific - **`CLIDocumentationGenerator`** — Main adapter class orchestrating documentation generation from the CLI. - `__init__(repo_path, output_dir, config, verbose=False, generate_html=False, diagrams_dir=None)` — Initializes job metadata, `ProgressTracker`, and `DocumentationJob`; configures backend logging. - `_configure_backend_logging()` — Sets up `logging.getLogger('codewiki.src.be')` with a `ColoredFormatter`; verbose mode logs `INFO`+ to stdout, non-verbose suppresses to `WARNING`+ on stderr. Disables propagation to avoid duplicate log lines. - - `generate()` — Main entry point. Builds a `BackendConfig` from CLI options (per-provider models, API keys, base URLs, token limits, temperature settings, clustering parameters, additional source paths), runs backend generation via `asyncio.run`, optionally triggers HTML generation, finalizes the job, and returns the completed `DocumentationJob`. Raises `APIError` on LLM failures. + - `generate()` — Main entry point. Builds a `BackendConfig` from CLI options (per-provider models, API keys, base URLs, token limits, temperature settings, clustering parameters), runs backend generation via `asyncio.run`, optionally triggers HTML generation, finalizes the job, and returns the completed `DocumentationJob`. Raises `APIError` on LLM failures. - `_run_backend_generation(backend_config)` — Async method driving the multi-stage backend pipeline (dependency analysis, clustering, doc generation) with `ProgressTracker` updates per stage. ## Usage Example @@ -24,7 +24,6 @@ config = { "main_api_key": "sk-...", "cluster_api_key": "sk-...", "max_depth": 2, - "additional_paths": ["shared/lib"], } generator = CLIDocumentationGenerator( @@ -42,5 +41,4 @@ print(job.status, job.output_directory) ## Notes - Requires `set_cli_context(True)` from `codewiki.src.config` to signal CLI mode to the backend. -- Additional source paths supplied in `config['additional_paths']` are resolved to absolute paths relative to `repo_path` before being passed to `BackendConfig`. - Verbose mode prints a detailed configuration summary (models, base URLs, token limits, clustering settings) prior to generation. \ No newline at end of file diff --git a/codewiki/cli/adapters/doc_generator.py b/codewiki/cli/adapters/doc_generator.py index ff93642a..f816d981 100644 --- a/codewiki/cli/adapters/doc_generator.py +++ b/codewiki/cli/adapters/doc_generator.py @@ -130,24 +130,6 @@ def generate(self) -> DocumentationJob: # Set CLI context for backend set_cli_context(True) - # Process additional paths from config - additional_paths_raw = self.config.get('additional_paths', []) - additional_paths_normalized = None - - if additional_paths_raw: - logger = logging.getLogger(__name__) - logger.info(f"🔍 CLI received {len(additional_paths_raw)} additional path(s):") - - additional_paths_normalized = [] - for i, path in enumerate(additional_paths_raw, 1): - # Convert to absolute path (relative to repo_path) - normalized = str((Path(self.repo_path) / path).resolve()) - logger.info(f" ├─ Path #{i}: {path}") - logger.info(f" │ └─ Normalized: {normalized}") - additional_paths_normalized.append(normalized) - - logger.info(f" └─ Normalized {len(additional_paths_normalized)} additional path(s)") - # Create backend config with CLI settings backend_config = BackendConfig.from_cli( repo_path=str(self.repo_path), @@ -190,9 +172,7 @@ def generate(self) -> DocumentationJob: # Agent instructions agent_instructions=self.config.get('agent_instructions'), # Optional diagrams directory - diagrams_dir=str(self.diagrams_dir) if self.diagrams_dir else None, - # Additional paths for dependency analysis (normalized) - additional_source_paths=additional_paths_normalized + diagrams_dir=str(self.diagrams_dir) if self.diagrams_dir else None ) # Log configuration details in verbose mode @@ -212,10 +192,6 @@ def generate(self) -> DocumentationJob: print(f" │ ├─ Max tokens/module: {backend_config.max_token_per_module}") print(f" │ ├─ Max tokens/leaf: {backend_config.max_token_per_leaf_module}") print(f" │ └─ Max depth: {backend_config.max_depth}") - if backend_config.additional_source_paths: - print(f" ├─ Additional Paths ({len(backend_config.additional_source_paths)}):") - for path in backend_config.additional_source_paths: - print(f" │ └─ {path}") if backend_config.agent_instructions: print(f" └─ Agent Instructions: Configured") if backend_config.agent_instructions.get('custom_instructions'): @@ -263,11 +239,6 @@ async def _run_backend_generation(self, backend_config: BackendConfig): print(f" ├─ Repository: {backend_config.repo_path}") print(f" └─ Output: {backend_config.output_dir}") - if backend_config.additional_source_paths: - print(f" ├─ Additional paths: {len(backend_config.additional_source_paths)}") - for i, path in enumerate(backend_config.additional_source_paths, 1): - print(f" │ └─ Path #{i}: {path}") - # Create documentation generator if self.verbose: logger.info("├─ Initializing DocumentationGenerator...") diff --git a/codewiki/cli/commands/generate.py b/codewiki/cli/commands/generate.py index ba469ae7..76903e63 100644 --- a/codewiki/cli/commands/generate.py +++ b/codewiki/cli/commands/generate.py @@ -138,12 +138,6 @@ def parse_patterns(patterns_str: str) -> List[str]: default=None, help="Maximum depth for hierarchical decomposition (overrides config)", ) -@click.option( - "--additional-paths", - type=str, - default=None, - help="Comma-separated additional paths to include in dependency analysis (e.g., 'vendor/deps,external/libs')", -) @click.pass_context def generate_command( ctx, @@ -162,8 +156,7 @@ def generate_command( max_tokens: Optional[int], max_token_per_module: Optional[int], max_token_per_leaf_module: Optional[int], - max_depth: Optional[int], - additional_paths: Optional[str] + max_depth: Optional[int] ): """ Generate comprehensive documentation for a code repository. @@ -208,10 +201,6 @@ def generate_command( \b # Override max depth for hierarchical decomposition $ codewiki generate --max-depth 3 - - \b - # Include additional dependency paths (e.g., vendor, external libs) - $ codewiki generate --additional-paths "vendor/packages,external/deps" """ logger = create_logger(verbose=verbose) start_time = time.time() @@ -377,43 +366,6 @@ def generate_command( if instructions: logger.debug(f"Custom instructions: {instructions}") - # Validate and parse additional paths - additional_paths_list = None - if additional_paths: - if verbose: - logger.debug("🔍 Stage 1.3: Additional Paths Validation") - - additional_paths_list = parse_patterns(additional_paths) - - if verbose: - logger.debug(f"├─ Parsed {len(additional_paths_list)} additional path(s)") - for i, path in enumerate(additional_paths_list, 1): - logger.debug(f"│ ├─ Path #{i}: {path}") - - # Validate that paths exist - invalid_paths = [] - for path in additional_paths_list: - full_path = repo_path / path - if not full_path.exists(): - invalid_paths.append(path) - elif verbose: - logger.debug(f"│ └─ ✓ {path} (exists)") - - if invalid_paths: - if verbose: - for path in invalid_paths: - logger.debug(f"│ └─ ✗ {path} (not found)") - raise RepositoryError( - f"Additional paths not found:\n" - f" {', '.join(invalid_paths)}\n\n" - f"Please ensure all paths exist relative to repository root:\n" - f" Repository: {repo_path}\n" - f" Invalid paths: {invalid_paths}" - ) - - if verbose: - logger.debug(f"└─ All additional paths validated") - # Log max token settings if verbose (from upstream) if verbose: effective_max_tokens = max_tokens if max_tokens is not None else config.main_max_tokens @@ -424,8 +376,6 @@ def generate_command( logger.debug(f"Max token/module: {effective_max_token_per_module}") logger.debug(f"Max token/leaf module: {effective_max_token_per_leaf}") logger.debug(f"Max depth: {effective_max_depth}") - if additional_paths_list: - logger.debug(f"Additional paths: {additional_paths_list}") # Get agent instructions (merge runtime with persistent) (from upstream) agent_instructions_dict = None @@ -485,8 +435,6 @@ def generate_command( 'max_depth': max_depth if max_depth is not None else config.max_depth, # Agent instructions 'agent_instructions': agent_instructions_dict, - # Additional paths for dependency analysis - 'additional_paths': additional_paths_list, }, verbose=verbose, generate_html=github_pages, diff --git a/codewiki/src/.config.md b/codewiki/src/.config.md index 0f012db6..5304efae 100644 --- a/codewiki/src/.config.md +++ b/codewiki/src/.config.md @@ -1,16 +1,14 @@ ## config.py -Defines the central `Config` dataclass that drives the CodeWiki documentation-generation pipeline, encapsulating repository paths, output directories, per-provider LLM settings (models, API keys, base URLs, temperatures, token limits), and agent instruction customization (include/exclude patterns, focus modules, doc type, custom instructions). It also supports multi-path source analysis and safe serialization that excludes runtime secrets. +Defines the central `Config` dataclass that drives the CodeWiki documentation-generation pipeline, encapsulating repository paths, output directories, per-provider LLM settings (models, API keys, base URLs, temperatures, token limits), and agent instruction customization (include/exclude patterns, focus modules, doc type, custom instructions). It also validates the repository path and supports safe serialization that excludes runtime secrets. ## Key Components -- **`Config`** — Core dataclass holding repo/output paths, per-provider (`cluster`, `main`, `fallback`) model/API/temperature/token settings, `agent_instructions`, and optional `additional_source_paths` for multi-path mode. +- **`Config`** — Core dataclass holding repo/output paths, per-provider (`cluster`, `main`, `fallback`) model/API/temperature/token settings, and `agent_instructions`. - **`Config.to_dict()` / `Config.from_dict()`** — Serialize/deserialize a `Config`, excluding API key fields (`cluster_api_key`, `main_api_key`, `fallback_api_key`) by default via `_RUNTIME_ONLY_SECRET_FIELDS` to prevent leaking secrets into logs/caches. - **Properties** — `include_patterns`, `exclude_patterns`, `focus_modules`, `doc_type`, `custom_instructions`: convenience accessors reading from `agent_instructions`. -- **`all_source_paths`** — Returns absolute paths for `repo_path` plus any `additional_source_paths`. -- **`validate_source_paths()`** — Verifies primary and additional paths exist, are directories, and are readable; raises `ValueError`/`OSError` otherwise. -- **`is_multi_path_mode()`** — Returns whether multiple source paths are configured. +- **`validate_repo_path()`** — Verifies the repository path exists and is a directory; raises `ValueError` otherwise. - **`get_prompt_addition()`** — Builds LLM prompt text based on `doc_type` and other agent instructions. - **Constants** — `OUTPUT_BASE_DIR`, `DEPENDENCY_GRAPHS_DIR`, `DOCS_DIR`, `MAX_DEPTH`, default token limits (`DEFAULT_MAX_TOKENS`, `DEFAULT_MAX_TOKEN_PER_MODULE`, `DEFAULT_MAX_TOKEN_PER_LEAF_MODULE`), and filenames (`MODULE_TREE_FILENAME`, `OVERVIEW_FILENAME`). - **`set_cli_context()` / `is_cli_context()`** — Track whether the module is running in CLI vs. web-app context, affecting how LLM credentials are loaded. @@ -34,12 +32,8 @@ config = Config( fallback_api_key="sk-...", ) -# Validate all configured source paths exist and are readable -config.validate_source_paths() - -# Check multi-path mode -if config.is_multi_path_mode(): - print("Analyzing paths:", config.all_source_paths) +# Validate the repository path exists and is a directory +config.validate_repo_path() # Serialize safely (no secrets included) safe_dict = config.to_dict() diff --git a/codewiki/src/be/agent_orchestrator.py b/codewiki/src/be/agent_orchestrator.py index 5a2113b3..d3df8122 100644 --- a/codewiki/src/be/agent_orchestrator.py +++ b/codewiki/src/be/agent_orchestrator.py @@ -50,6 +50,7 @@ from codewiki.src.be.agent_tools.deps import CodeWikiDeps from codewiki.src.be.agent_tools.read_code_components import read_code_components_tool from codewiki.src.be.agent_tools.str_replace_editor import str_replace_editor_tool +from codewiki.src.be.agent_tools.code_graph import code_graph_tools from codewiki.src.be.agent_tools.generate_sub_module_documentations import generate_sub_module_documentation_tool from codewiki.src.be.llm_services import create_fallback_models, reset_request_counter from codewiki.src.be.prompt_template import ( @@ -95,7 +96,8 @@ def create_agent(self, module_name: str, components: Dict[str, Any], tools=[ read_code_components_tool, str_replace_editor_tool, - generate_sub_module_documentation_tool + generate_sub_module_documentation_tool, + *code_graph_tools(), ], system_prompt=format_system_prompt(module_name, self.custom_instructions), retries=3, # From fork @@ -105,7 +107,7 @@ def create_agent(self, module_name: str, components: Dict[str, Any], self.fallback_models, name=module_name, deps_type=CodeWikiDeps, - tools=[read_code_components_tool, str_replace_editor_tool], + tools=[read_code_components_tool, str_replace_editor_tool, *code_graph_tools()], system_prompt=format_leaf_system_prompt(module_name, self.custom_instructions), retries=3, # From fork ) diff --git a/codewiki/src/be/agent_tools/code_graph.py b/codewiki/src/be/agent_tools/code_graph.py new file mode 100644 index 00000000..f9085210 --- /dev/null +++ b/codewiki/src/be/agent_tools/code_graph.py @@ -0,0 +1,193 @@ +"""Code-graph agent tools served by the Flamingo hub. + +CodeWiki parses ONLY the repository it documents. Code that repository +imports from another repository is not on disk; the writing agents look it +up on demand through the hub's code-graph tools over HTTP: + +- ``GET {HUB_BASE_URL}/api/ci/code-documentation/tools?github=`` + lists the tools this repository may call (the set differs between public + and private repositories, so it is never hard-coded here), each with its + name, description and JSON Schema for its arguments. +- ``POST {HUB_BASE_URL}/api/ci/code-documentation/tools`` with + ``{"github", "tool", "args"}`` runs one and answers ``{"content": str}``. + The content is wrapped by the hub in a ``<<``. +Without ``HUB_BASE_URL``, ``WEBHOOK_SECRET`` or ``GITHUB_REPOSITORY``, or +when the list cannot be read, no graph tool is offered and documentation +runs on the repository alone. A failed call is reported to the model as a +failed tool result (``ToolFailed``), never raised into the agent run. +""" + +from __future__ import annotations + +import logging +import os +from typing import Any, Dict, List, Optional +from urllib.parse import urlparse + +import httpx +from pydantic_ai import Tool +from pydantic_ai.exceptions import ToolFailed + +logger = logging.getLogger(__name__) + +TOOLS_PATH = "/api/ci/code-documentation/tools" +LIST_TIMEOUT_SECONDS = 30.0 +CALL_TIMEOUT_SECONDS = 60.0 +LOOPBACK_HOSTS = {"localhost", "127.0.0.1"} + +# Names of CodeWiki's own agent tools: a hub tool never replaces one. +LOCAL_TOOL_NAMES = {"read_code_components", "str_replace_editor", "generate_sub_module_documentation"} + +_cached_tools: Optional[List[Tool]] = None + + +class _HubSettings: + """What the hub calls need, read from the environment once.""" + + def __init__(self, base_url: str, secret: str, repository: str): + self.base_url = base_url.rstrip("/") + self.secret = secret + self.repository = repository + + @property + def endpoint(self) -> str: + return f"{self.base_url}{TOOLS_PATH}" + + @property + def headers(self) -> Dict[str, str]: + return {"Authorization": f"Bearer {self.secret}", "Accept": "application/json"} + + +def _read_settings() -> Optional[_HubSettings]: + base_url = (os.environ.get("HUB_BASE_URL") or "").strip() + secret = (os.environ.get("WEBHOOK_SECRET") or "").strip() + repository = (os.environ.get("GITHUB_REPOSITORY") or "").strip() + missing = [ + name + for name, value in ( + ("HUB_BASE_URL", base_url), + ("WEBHOOK_SECRET", secret), + ("GITHUB_REPOSITORY", repository), + ) + if not value + ] + if missing: + logger.info(f"Code-graph tools are off: {', '.join(missing)} not set") + return None + parsed = urlparse(base_url) + secure = parsed.scheme == "https" or (parsed.scheme == "http" and parsed.hostname in LOOPBACK_HOSTS) + if not secure or not parsed.hostname: + logger.warning( + "Code-graph tools are off: HUB_BASE_URL must be https " + "(http only for localhost or 127.0.0.1)" + ) + return None + return _HubSettings(base_url, secret, repository) + + +def _error_message(response: httpx.Response) -> str: + """The hub's `error` and `code`, else the HTTP status.""" + try: + body = response.json() + except ValueError: + body = None + if isinstance(body, dict) and (body.get("error") or body.get("code")): + code = body.get("code") or f"HTTP_{response.status_code}" + return f"{code}: {body.get('error') or 'request refused'}" + return f"HTTP_{response.status_code}: the hub refused the request" + + +def _make_tool(settings: _HubSettings, spec: Dict[str, Any]) -> Tool: + name = spec["name"] + + async def call_hub_tool(**args: Any) -> str: + payload = {"github": settings.repository, "tool": name, "args": args} + try: + async with httpx.AsyncClient(timeout=CALL_TIMEOUT_SECONDS) as client: + response = await client.post(settings.endpoint, json=payload, headers=settings.headers) + except httpx.TimeoutException: + raise ToolFailed(f"The code graph did not answer {name} within {int(CALL_TIMEOUT_SECONDS)} s. Continue without it.") + except httpx.HTTPError as error: + raise ToolFailed(f"The code graph could not be reached for {name} ({type(error).__name__}). Continue without it.") + if response.status_code >= 400: + message = _error_message(response) + logger.warning(f"Code-graph tool {name} failed: {message}") + raise ToolFailed(f"{name} failed: {message}") + try: + body = response.json() + except ValueError: + raise ToolFailed(f"{name} failed: the hub's answer was not JSON") + content = body.get("content") if isinstance(body, dict) else None + if not isinstance(content, str): + raise ToolFailed(f"{name} failed: the hub's answer had no content") + return content + + return Tool.from_schema( + function=call_hub_tool, + name=name, + description=spec.get("description") or None, + json_schema=spec["input_schema"], + ) + + +def _fetch_tools(settings: _HubSettings) -> List[Tool]: + try: + response = httpx.get( + settings.endpoint, + params={"github": settings.repository}, + headers=settings.headers, + timeout=LIST_TIMEOUT_SECONDS, + ) + except httpx.HTTPError as error: + logger.warning(f"Code-graph tools are off: the hub could not be reached ({type(error).__name__})") + return [] + if response.status_code >= 400: + logger.warning(f"Code-graph tools are off: listing them failed ({_error_message(response)})") + return [] + try: + body = response.json() + except ValueError: + logger.warning("Code-graph tools are off: the hub's tool list was not JSON") + return [] + specs = body.get("tools") if isinstance(body, dict) else None + if not isinstance(specs, list): + logger.warning("Code-graph tools are off: the hub's answer had no tool list") + return [] + + tools: List[Tool] = [] + for spec in specs: + if not ( + isinstance(spec, dict) + and isinstance(spec.get("name"), str) + and spec["name"] + and isinstance(spec.get("input_schema"), dict) + ): + logger.warning("Skipping a code-graph tool the hub described without a name or input schema") + continue + if spec["name"] in LOCAL_TOOL_NAMES: + logger.warning(f"Skipping code-graph tool {spec['name']}: CodeWiki has a tool of that name") + continue + tools.append(_make_tool(settings, spec)) + logger.info( + f"Code-graph tools on for {settings.repository}: " + f"{', '.join(tool.name for tool in tools) or 'none served'}" + ) + return tools + + +def code_graph_tools() -> List[Tool]: + """The hub's code-graph tools for this repository, read once per process.""" + global _cached_tools + if _cached_tools is None: + settings = _read_settings() + _cached_tools = _fetch_tools(settings) if settings else [] + return list(_cached_tools) + + +def reset_code_graph_tools_cache() -> None: + """Forget the tool list read from the hub (tests).""" + global _cached_tools + _cached_tools = None diff --git a/codewiki/src/be/agent_tools/generate_sub_module_documentations.py b/codewiki/src/be/agent_tools/generate_sub_module_documentations.py index aa4272e8..41b9598a 100644 --- a/codewiki/src/be/agent_tools/generate_sub_module_documentations.py +++ b/codewiki/src/be/agent_tools/generate_sub_module_documentations.py @@ -22,6 +22,7 @@ from codewiki.src.be.agent_tools.deps import CodeWikiDeps from codewiki.src.be.agent_tools.read_code_components import read_code_components_tool from codewiki.src.be.agent_tools.str_replace_editor import str_replace_editor_tool +from codewiki.src.be.agent_tools.code_graph import code_graph_tools from codewiki.src.be.llm_services import create_fallback_models from codewiki.src.be.prompt_template import SYSTEM_PROMPT, LEAF_SYSTEM_PROMPT, format_user_prompt, format_system_prompt, format_leaf_system_prompt from codewiki.src.be.utils import is_complex_module, count_tokens @@ -117,7 +118,7 @@ async def generate_sub_module_documentation( name=sub_module_name, deps_type=CodeWikiDeps, system_prompt=format_system_prompt(sub_module_name, ctx.deps.custom_instructions), - tools=[read_code_components_tool, str_replace_editor_tool, generate_sub_module_documentation_tool], + tools=[read_code_components_tool, str_replace_editor_tool, generate_sub_module_documentation_tool, *code_graph_tools()], retries=3, ) else: @@ -126,7 +127,7 @@ async def generate_sub_module_documentation( name=sub_module_name, deps_type=CodeWikiDeps, system_prompt=format_leaf_system_prompt(sub_module_name, ctx.deps.custom_instructions), - tools=[read_code_components_tool, str_replace_editor_tool], + tools=[read_code_components_tool, str_replace_editor_tool, *code_graph_tools()], retries=3, ) diff --git a/codewiki/src/be/agent_tools/str_replace_editor.py b/codewiki/src/be/agent_tools/str_replace_editor.py index b0319b4b..cc5da287 100644 --- a/codewiki/src/be/agent_tools/str_replace_editor.py +++ b/codewiki/src/be/agent_tools/str_replace_editor.py @@ -11,7 +11,6 @@ from collections import defaultdict from pathlib import Path from typing import List, Optional, Tuple, Literal -import io import logging @@ -26,9 +25,11 @@ # There are some super strange "ascii can't decode x" errors, -# that can be solved with setting the default encoding for stdout -# (note that python3.6 doesn't have the reconfigure method) -sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8") +# that can be solved with setting the default encoding for stdout. +# Reconfigure the stream in place: wrapping its buffer in a new TextIOWrapper +# closed the original stream when the wrapper was collected. +if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(encoding="utf-8") TRUNCATED_MESSAGE: str = "To save on context only part of this file has been shown to you. You should retry this tool after you have searched inside the file with `grep -n` in order to find the line numbers of what you are looking for." MAX_RESPONSE_LEN: int = 16000 diff --git a/codewiki/src/be/dependency_analyzer/.ast_parser.md b/codewiki/src/be/dependency_analyzer/.ast_parser.md index 1caf398a..1cf964d7 100644 --- a/codewiki/src/be/dependency_analyzer/.ast_parser.md +++ b/codewiki/src/be/dependency_analyzer/.ast_parser.md @@ -1,16 +1,14 @@ ## ast_parser.py -This module implements Stage 2 of the dependency analysis pipeline: parsing one or more repositories into Abstract Syntax Tree (AST)-derived structural and call-graph representations, then building a namespaced, fully-qualified dependency graph of code components (functions, classes, etc.) suitable for downstream clustering, summarization, and documentation generation. +This module implements Stage 2 of the dependency analysis pipeline: parsing the repository into Abstract Syntax Tree (AST)-derived structural and call-graph representations, then building a namespaced, fully-qualified dependency graph of code components (functions, classes, etc.) suitable for downstream clustering, summarization, and documentation generation. ## Key Components -- **`DependencyParser`**: Main class orchestrating repository parsing. Supports both single-path and multi-path (multiple repository) modes. - - `parse_repository(filtered_folders=None)`: Entry point that dispatches to single- or multi-repository parsing based on the number of paths provided. - - `_parse_single_repository(repo_path, filtered_folders=None)`: Analyzes structure and call graph for one repository, builds components without namespacing. - - `_parse_multiple_repositories(filtered_folders=None)`: Analyzes each repository, namespaces its components (using directory name as prefix), merges all components, and resolves cross-namespace dependency edges. - - `_get_namespace_from_path(repo_path)`: Derives a namespace string from a repository's directory name. - - `_build_namespaced_components(call_graph_result, namespace, namespace_mapping, repo_index)`: Constructs `Node` objects keyed by fully-qualified domain name (FQDN) in `namespace.original_id` format, tracking `short_id`, `namespace`, and `is_from_deps` metadata. +- **`DependencyParser`**: Main class orchestrating repository parsing. Only the repository itself is parsed; code it imports from other repositories is looked up by the writing agents through the hub's code-graph tools. + - `parse_repository(filtered_folders=None)`: Analyzes structure and call graph for the repository and builds its components. + - `_get_namespace_from_path(repo_path)`: Derives a namespace string from the repository's directory name. + - `_build_components_from_analysis(call_graph_result)`: Constructs `Node` objects keyed by fully-qualified domain name (FQDN) in `namespace.original_id` format, tracking `short_id` and `namespace` metadata. - **`Node`** (imported from `codewiki.src.be.dependency_analyzer.models.core`): Data model representing a single code component (function, class, etc.) with metadata such as source location, docstring, parameters, and FQDN identifiers. @@ -21,7 +19,6 @@ This module implements Stage 2 of the dependency analysis pipeline: parsing one ```python from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser -# Single repository mode parser = DependencyParser( repo_path="/path/to/repo", include_patterns=["*.py", "*.cs"], @@ -29,12 +26,6 @@ parser = DependencyParser( ) components = parser.parse_repository() -# Multi-repository mode (e.g., main app + dependency) -parser = DependencyParser( - repo_path=["/path/to/main-app", "/path/to/shared-lib"] -) -components = parser.parse_repository() - # Inspect resulting components for fqdn, node in components.items(): print(fqdn, node.component_type, node.namespace) diff --git a/codewiki/src/be/dependency_analyzer/.dependency_graphs_builder.md b/codewiki/src/be/dependency_analyzer/.dependency_graphs_builder.md index 8f792aaa..a9edaafb 100644 --- a/codewiki/src/be/dependency_analyzer/.dependency_graphs_builder.md +++ b/codewiki/src/be/dependency_analyzer/.dependency_graphs_builder.md @@ -6,10 +6,10 @@ This module provides the `DependencyGraphBuilder` class, which orchestrates the ## Key Components - **`DependencyGraphBuilder`** — Main class that drives dependency analysis. - - `__init__(config: Config)` — Initializes the builder with a `Config` object containing repo paths, include/exclude patterns, and output directories. + - `__init__(config: Config)` — Initializes the builder with a `Config` object containing the repository path, include/exclude patterns, and output directories. - `build_dependency_graph() -> tuple[Dict[str, Any], List[str]]` — Core method that: 1. Ensures the output directory exists and computes sanitized file paths for saving results. - 2. Supports single-path or multi-path repository analysis via `Config.all_source_paths`. + 2. Analyzes the repository at `Config.repo_path` only. 3. Uses `DependencyParser` to parse source files into components (respecting include/exclude patterns). 4. Logs a breakdown of parsed component types. 5. Saves the dependency graph to disk via `parser.save_dependency_graph`. diff --git a/codewiki/src/be/dependency_analyzer/ast_parser.py b/codewiki/src/be/dependency_analyzer/ast_parser.py index cc32defb..88848288 100644 --- a/codewiki/src/be/dependency_analyzer/ast_parser.py +++ b/codewiki/src/be/dependency_analyzer/ast_parser.py @@ -1,13 +1,14 @@ -"""AST parsing and dependency graph construction for multi-repository codebases. +"""AST parsing and dependency graph construction for a repository. This module implements the core dependency analysis pipeline stage that: -- Parses one or more repositories (single-path or multi-path modes) into - structural and call-graph representations using the AnalysisService. +- Parses the repository into structural and call-graph representations + using the AnalysisService. Only the repository itself is parsed: code it + imports from other repositories is looked up on demand by the writing + agents through the hub's code-graph tools (see agent_tools/code_graph.py). - Builds Node-based components keyed by fully-qualified domain names (FQDNs) - in the canonical `module.path::ComponentName` format. -- Namespaces components originating from multiple repositories to avoid ID - collisions and tracks module membership for each component. -- Resolves intra- and cross-namespace dependency edges between components. + in the canonical `module.path::ComponentName` format, namespaced by the + repository's directory name, and tracks module membership for each one. +- Resolves dependency edges between components. - Persists the resulting dependency graph to disk for downstream consumers (e.g., clustering, LLM-based summarization, and documentation generation). """ @@ -16,7 +17,7 @@ import logging import argparse from dataclasses import dataclass, field -from typing import Dict, List, Set, Tuple, Optional, Any, Union +from typing import Dict, List, Set, Tuple, Optional, Any from pathlib import Path import re @@ -28,24 +29,18 @@ class DependencyParser: - """Parser for extracting code components from multi-language repositories.""" + """Parser for extracting code components from a multi-language repository.""" - def __init__(self, repo_path: Union[str, List[str]], include_patterns: List[str] = None, exclude_patterns: List[str] = None): + def __init__(self, repo_path: str, include_patterns: List[str] = None, exclude_patterns: List[str] = None): """ Initialize the dependency parser. Args: - repo_path: Path to the repository or list of paths to multiple repositories + repo_path: Path to the repository include_patterns: File patterns to include (e.g., ["*.cs", "*.py"]) exclude_patterns: File/directory patterns to exclude (e.g., ["*Tests*"]) """ - # Support both single path (str) and multiple paths (List[str]) - if isinstance(repo_path, str): - self.repo_paths = [os.path.abspath(repo_path)] - self.repo_path = self.repo_paths[0] # Maintain backward compatibility - else: - self.repo_paths = [os.path.abspath(p) for p in repo_path] - self.repo_path = self.repo_paths[0] # Default to first path for backward compat + self.repo_path = os.path.abspath(repo_path) self.components: Dict[str, Node] = {} self.modules: Set[str] = set() @@ -56,8 +51,7 @@ def __init__(self, repo_path: Union[str, List[str]], include_patterns: List[str] def parse_repository(self, filtered_folders: List[str] = None) -> Dict[str, Node]: """ - Parse repository or repositories and extract components. - If multiple paths were provided, components are namespaced by source directory. + Parse the repository and extract components. Args: filtered_folders: Optional list of folders to filter @@ -65,20 +59,8 @@ def parse_repository(self, filtered_folders: List[str] = None) -> Dict[str, Node Returns: Dictionary mapping component IDs to Node objects """ + repo_path = self.repo_path logger.info("🔍 Stage 2: AST Parsing & Dependency Analysis") - logger.info(f"├─ Repository paths: {len(self.repo_paths)}") - - if len(self.repo_paths) == 1: - logger.info(f"│ └─ Single-path mode: {self.repo_paths[0]}") - # Single path - maintain backward compatibility - return self._parse_single_repository(self.repo_paths[0], filtered_folders) - else: - logger.info(f"│ └─ Multi-path mode: {len(self.repo_paths)} sources") - # Multiple paths - use multi-path parsing - return self._parse_multiple_repositories(filtered_folders) - - def _parse_single_repository(self, repo_path: str, filtered_folders: List[str] = None) -> Dict[str, Node]: - """Parse a single repository (backward compatible).""" logger.info(f"📂 Parsing repository: {repo_path}") # Log custom patterns if set @@ -111,89 +93,6 @@ def _parse_single_repository(self, repo_path: str, filtered_folders: List[str] = logger.info(f" └─ Total modules: {len(self.modules)}") return self.components - def _parse_multiple_repositories(self, filtered_folders: List[str] = None) -> Dict[str, Node]: - """ - Parse multiple repositories and merge components with namespace prefixes. - - Each repository gets a namespace prefix based on its directory name. - Component IDs are prefixed to avoid collisions: {namespace}::{original_id} - - Returns: - Dictionary of all components from all repositories with namespaced IDs - """ - logger.info(f"🔍 Multi-path mode detected: analyzing {len(self.repo_paths)} source directories") - logger.info(f" ├─ Primary: {self.repo_paths[0]}") - for i, path in enumerate(self.repo_paths[1:], 1): - logger.info(f" └─ Additional #{i}: {path}") - - all_components: Dict[str, Node] = {} - namespace_mapping: Dict[str, str] = {} # Maps original IDs to namespaced IDs - - for idx, repo_path in enumerate(self.repo_paths): - # Create namespace from directory name - namespace = self._get_namespace_from_path(repo_path) - logger.info(f"\n📂 Analyzing path {idx+1}/{len(self.repo_paths)}: {repo_path}") - logger.info(f" └─ Namespace: '{namespace}'") - - # Log custom patterns if set - if self.include_patterns: - logger.info(f"Using custom include patterns: {self.include_patterns}") - if self.exclude_patterns: - logger.info(f"Using custom exclude patterns: {self.exclude_patterns}") - - # Analyze structure - structure_result = self.analysis_service._analyze_structure( - repo_path, - include_patterns=self.include_patterns, - exclude_patterns=self.exclude_patterns - ) - - # Analyze call graph - call_graph_result = self.analysis_service._analyze_call_graph( - structure_result["file_tree"], - repo_path - ) - - # Build components with namespace (idx=0 is main repo, idx>0 is deps) - repo_components = self._build_namespaced_components( - call_graph_result, - namespace, - namespace_mapping, - repo_index=idx - ) - - # Merge into all_components - all_components.update(repo_components) - - logger.info(f"📊 Namespace '{namespace}': found {len(repo_components)} components") - - # Log component type breakdown for this namespace - type_counts = {} - for comp in repo_components.values(): - comp_type = comp.component_type - type_counts[comp_type] = type_counts.get(comp_type, 0) + 1 - - if type_counts: - logger.info(f" └─ Component types:") - for comp_type, count in sorted(type_counts.items(), key=lambda x: x[1], reverse=True): - logger.info(f" • {comp_type}: {count}") - - # Update cross-repository dependencies - logger.info(f"\n🔗 Resolving cross-namespace dependencies...") - cross_deps_count = self._resolve_cross_namespace_dependencies(all_components, namespace_mapping) - if cross_deps_count > 0: - logger.info(f" └─ ✓ Resolved {cross_deps_count} cross-namespace dependencies") - else: - logger.info(f" └─ No cross-namespace dependencies found") - - self.components = all_components - logger.info(f"\n📊 Multi-path analysis complete:") - logger.info(f" ├─ Total components: {len(all_components)}") - logger.info(f" ├─ Total modules: {len(self.modules)}") - logger.info(f" └─ Namespaces: {len(self.repo_paths)}") - - return self.components - def _get_namespace_from_path(self, repo_path: str) -> str: """ Generate a namespace prefix from a repository path. @@ -202,151 +101,11 @@ def _get_namespace_from_path(self, repo_path: str) -> str: repo_path: Absolute path to repository Returns: - Namespace string (e.g., 'openframe-frontend', 'ui-kit') + Namespace string (e.g., 'openframe-frontend') """ # Use the last directory name as namespace return os.path.basename(repo_path.rstrip(os.sep)) - def _build_namespaced_components( - self, - call_graph_result: Dict, - namespace: str, - namespace_mapping: Dict[str, str], - repo_index: int = 0 - ) -> Dict[str, Node]: - """ - Build components with namespace prefixes. - - Args: - call_graph_result: Result from call graph analysis - namespace: Namespace prefix for this repository - namespace_mapping: Global mapping of original IDs to namespaced IDs - repo_index: Index of this repository (0 = main, >0 = deps) - - Returns: - Dictionary of namespaced components - """ - functions = call_graph_result.get("functions", []) - relationships = call_graph_result.get("relationships", []) - - components: Dict[str, Node] = {} - - # First pass: Create components with namespaced IDs - for func_dict in functions: - original_id = func_dict.get("id", "") - if not original_id: - continue - - # Create FQDN (namespaced component ID) using '::' to separate - # the namespace/module path from the component identifier - fqdn = f"{namespace}.{original_id}" - - # Store mapping for dependency resolution - namespace_mapping[original_id] = fqdn - - # Create node with FQDN as id and metadata fields - node = Node( - id=fqdn, # FQDN as primary identifier - name=func_dict.get("name", ""), - component_type=func_dict.get("component_type", func_dict.get("node_type", "function")), - file_path=func_dict.get("file_path", ""), - relative_path=func_dict.get("relative_path", ""), - source_code=func_dict.get("source_code", func_dict.get("code_snippet", "")), - start_line=func_dict.get("start_line", 0), - end_line=func_dict.get("end_line", 0), - has_docstring=func_dict.get("has_docstring", bool(func_dict.get("docstring", ""))), - docstring=func_dict.get("docstring", "") or "", - parameters=func_dict.get("parameters", []), - node_type=func_dict.get("node_type", "function"), - base_classes=func_dict.get("base_classes"), - class_name=func_dict.get("class_name"), - display_name=func_dict.get("display_name", ""), - component_id=fqdn, - # FQDN metadata fields - short_id=original_id, - namespace=namespace, - is_from_deps=(repo_index > 0) - ) - - # Dictionary key is FQDN - components[fqdn] = node - - # Track module (with namespace) - # original_id is '::' from the analyzers, so the - # module path is everything before '::'. (The dot-split fallback is - # for ids that predate the '::' separator.) - module_path = ( - original_id.split("::")[0] - if "::" in original_id - else ".".join(original_id.split(".")[:-1]) - ) - if module_path: - self.modules.add(f"{namespace}.{module_path}") - - # Second pass: Add dependencies within this namespace - for rel_dict in relationships: - caller_id = rel_dict.get("caller", "") - callee_id = rel_dict.get("callee", "") - - # Map to namespaced IDs - namespaced_caller = namespace_mapping.get(caller_id) - namespaced_callee = namespace_mapping.get(callee_id) - - if namespaced_caller and namespaced_caller in components: - if namespaced_callee: - components[namespaced_caller].depends_on.add(namespaced_callee) - - return components - - def _resolve_cross_namespace_dependencies( - self, - all_components: Dict[str, Node], - namespace_mapping: Dict[str, str] - ) -> int: - """ - Resolve dependencies that cross namespace boundaries. - - This handles cases where one component in namespace A depends on - a component in namespace B. - - Args: - all_components: All components from all repositories - namespace_mapping: Mapping of original IDs to namespaced IDs - - Returns: - Number of cross-namespace dependencies resolved - """ - cross_deps_resolved = 0 - - # For each component, check if any dependencies need cross-namespace resolution - for component_id, component in sorted(all_components.items()): # ✅ SORT for determinism - resolved_deps = set() - - for dep_id in component.depends_on: - # If dependency exists in all_components, it's already resolved - if dep_id in all_components: - resolved_deps.add(dep_id) - else: - # Try to find it by name match across namespaces - dep_name = dep_id.split(".")[-1] - for other_id, other_component in sorted(all_components.items()): # ✅ SORT for determinism - if other_component.name == dep_name and other_id != component_id: - # Extract namespaces to check if it's cross-namespace - source_namespace = component_id.split(".")[0] - target_namespace = other_id.split(".")[0] - if source_namespace != target_namespace: - logger.debug(f" ├─ Cross-namespace dependency: {component_id} → {other_id}") - cross_deps_resolved += 1 - resolved_deps.add(other_id) - break - else: - # Keep original if not resolved - resolved_deps.add(dep_id) - - component.depends_on = resolved_deps - - return cross_deps_resolved - def _build_components_from_analysis(self, call_graph_result: Dict): functions = call_graph_result.get("functions", []) relationships = call_graph_result.get("relationships", []) @@ -383,8 +142,7 @@ def _build_components_from_analysis(self, call_graph_result: Dict): component_id=fqdn, # FQDN metadata fields short_id=original_id, - namespace=namespace, - is_from_deps=False # Single-path mode = main repo + namespace=namespace ) # Dictionary key is FQDN diff --git a/codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py b/codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py index 96d7abe7..87d8847b 100644 --- a/codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py +++ b/codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py @@ -49,20 +49,9 @@ def build_dependency_graph(self) -> tuple[Dict[str, Any], List[str]]: include_patterns = self.config.include_patterns if self.config.include_patterns else None exclude_patterns = self.config.exclude_patterns if self.config.exclude_patterns else None - # Build list of paths to analyze - repo_paths = self.config.all_source_paths - - # Log multi-path mode if enabled - if self.config.is_multi_path_mode(): - logger.info(f"🔍 Multi-path mode enabled: analyzing {len(repo_paths)} source paths") - logger.info(f" ├─ Primary: {repo_paths[0]}") - for i, path in enumerate(repo_paths[1:], 1): - logger.info(f" └─ Additional #{i}: {path}") - else: - logger.info(f"📁 Single-path mode: analyzing {repo_paths[0]}") - + logger.info(f"📁 Analyzing repository: {self.config.repo_path}") parser = DependencyParser( - repo_paths if len(repo_paths) > 1 else repo_paths[0], + self.config.repo_path, include_patterns=include_patterns, exclude_patterns=exclude_patterns ) diff --git a/codewiki/src/be/dependency_analyzer/models/.core.md b/codewiki/src/be/dependency_analyzer/models/.core.md index cb41a6f3..4da5a06e 100644 --- a/codewiki/src/be/dependency_analyzer/models/.core.md +++ b/codewiki/src/be/dependency_analyzer/models/.core.md @@ -5,7 +5,7 @@ Defines the core Pydantic data models used throughout the dependency analysis pi ## Key Components -- **`Node`**: Represents a single code entity (function, class, module, etc.) discovered during analysis. Uses a fully-qualified domain name (FQDN) format (`{namespace}.{original_id}`) for its `id` to disambiguate entities across the main repository and its dependencies. Tracks source location (`file_path`, `start_line`, `end_line`), documentation (`has_docstring`, `docstring`), structural metadata (`component_type`, `class_name`, `base_classes`, `parameters`), and outgoing dependencies (`depends_on`). Includes FQDN-related fields (`short_id`, `namespace`, `is_from_deps`) for display and provenance, all defaulted for backward compatibility. +- **`Node`**: Represents a single code entity (function, class, module, etc.) discovered during analysis. Uses a fully-qualified domain name (FQDN) format (`{namespace}.{original_id}`) for its `id`, namespaced by the repository's directory name. Tracks source location (`file_path`, `start_line`, `end_line`), documentation (`has_docstring`, `docstring`), structural metadata (`component_type`, `class_name`, `base_classes`, `parameters`), and outgoing dependencies (`depends_on`). Includes FQDN-related fields (`short_id`, `namespace`) for display and provenance, all defaulted for backward compatibility. - `get_display_name()`: Returns `display_name` if set, otherwise falls back to `name`. - **`CallRelationship`**: Represents a directed call-graph edge between two nodes, identified by `caller` and `callee` IDs. Optionally records the source `call_line` and whether the relationship was successfully `is_resolved` to a known node. @@ -26,7 +26,6 @@ node = Node( relative_path="utils.py", short_id="utils.parse_config", namespace="main", - is_from_deps=False, depends_on={"main.utils.load_file"}, ) diff --git a/codewiki/src/be/dependency_analyzer/models/core.py b/codewiki/src/be/dependency_analyzer/models/core.py index 3902c374..eea675ee 100644 --- a/codewiki/src/be/dependency_analyzer/models/core.py +++ b/codewiki/src/be/dependency_analyzer/models/core.py @@ -46,8 +46,7 @@ class Node(BaseModel): # FQDN metadata fields (all have defaults for backward compatibility) short_id: str = "" # Original ID without namespace (for display) - namespace: str = "" # Namespace (e.g., "main", "deps", "ui-kit") - is_from_deps: bool = False # True if from dependencies, False if from main repo + namespace: str = "" # Namespace: the repository's directory name def get_display_name(self) -> str: return self.display_name or self.name diff --git a/codewiki/src/be/prompt_template.py b/codewiki/src/be/prompt_template.py index dda93534..3fd6b8b8 100644 --- a/codewiki/src/be/prompt_template.py +++ b/codewiki/src/be/prompt_template.py @@ -11,6 +11,21 @@ _VALIDATION_RULES_SECTION = get_validation_rules_section() +# Code from other repositories is NOT parsed or copied: the writing agents look +# it up through the hub's code-graph tools (agent_tools/code_graph.py). Plain +# text with no braces, so it is safe inside the f-string prompts below. +_CODE_GRAPH_TOOLS_SECTION = """ +Code this repository imports from another repository is NOT on disk and is not among the components you can read. +When code-graph tools are available, look such code up instead of guessing: +- `resolve_code_imports`: what an import refers to and which repository defines it +- `get_code_symbol`: a definition, its signature and its callers +- `get_repo_ecosystem`: which artifacts and versions this repository uses +Document how THIS repository uses that code, never the other repository's internals. +Tool results arrive wrapped in a `<<""" + + def format_module_display_name(module_name: str) -> str: """ Convert kebab-case module names to Title Case for display. @@ -360,7 +375,9 @@ def format_module_display_name(module_name: str) -> str: - `str_replace_editor`: File system operations for creating and editing documentation files - `read_code_components`: Explore additional code dependencies not included in the provided components - `generate_sub_module_documentation`: Generate detailed documentation for individual sub-modules via sub-agents +- Code-graph tools (when available): look up code this repository imports from other repositories +{_CODE_GRAPH_TOOLS_SECTION} {{custom_instructions}} """.strip() @@ -552,7 +569,9 @@ def format_module_display_name(module_name: str) -> str: - `str_replace_editor`: File system operations for creating and editing documentation files - `read_code_components`: Explore additional code dependencies not included in the provided components +- Code-graph tools (when available): look up code this repository imports from other repositories +{_CODE_GRAPH_TOOLS_SECTION} {{custom_instructions}} """.strip() diff --git a/codewiki/src/config.py b/codewiki/src/config.py index bcd89551..f86045dc 100644 --- a/codewiki/src/config.py +++ b/codewiki/src/config.py @@ -10,7 +10,7 @@ arguments (`from_args`), for a web-app job (`from_web_job`), from explicit CLI parameters (`from_cli`), and from a `ConfigManager` (`from_config_manager`), along with helpers for -multi-path source validation and prompt-addition generation used by the +repository-path validation and prompt-addition generation used by the downstream documentation generation stages. """ @@ -113,10 +113,6 @@ class Config: # Optional separate directory for Mermaid diagrams (from fork) # When set, diagrams are extracted from markdown and saved as separate files diagrams_dir: str = None - # Multi-path support: additional source directories to analyze alongside repo_path - # When None, operates in single-path mode (backward compatible) - # When set, all paths are analyzed and merged into unified documentation - additional_source_paths: Optional[List[str]] = None def to_dict(self, include_secrets: bool = False) -> Dict[str, Any]: """ @@ -190,84 +186,23 @@ def custom_instructions(self) -> Optional[str]: return instructions_dict.get('custom_instructions') return None - @property - def all_source_paths(self) -> List[str]: - """ - Get all source paths to analyze (primary + additional). - - Returns: - List of absolute paths. Always includes repo_path as first element. - If additional_source_paths is None, returns single-element list. + def validate_repo_path(self) -> None: """ - import logging - logger = logging.getLogger(__name__) - - paths = [os.path.abspath(self.repo_path)] - if self.additional_source_paths: - paths.extend([os.path.abspath(p) for p in self.additional_source_paths]) - logger.debug(f"🔍 all_source_paths property accessed:") - logger.debug(f" ├─ Primary: {paths[0]}") - for i, path in enumerate(paths[1:], 1): - logger.debug(f" └─ Additional #{i}: {path}") - else: - logger.debug(f"🔍 all_source_paths property accessed (single-path mode): {paths[0]}") - return paths - - def validate_source_paths(self) -> None: - """ - Validate that all source paths exist and are accessible directories. + Validate that the repository path exists and is a directory. Raises: - ValueError: If any path does not exist or is not a directory - OSError: If any path cannot be accessed + ValueError: If the path does not exist or is not a directory """ import logging logger = logging.getLogger(__name__) - logger.info("🔍 Validating source paths...") + logger.info("🔍 Validating repository path...") - # Validate primary repo_path - logger.debug(f" ├─ Checking primary path: {self.repo_path}") if not os.path.exists(self.repo_path): - raise ValueError(f"Primary repository path does not exist: {self.repo_path}") + raise ValueError(f"Repository path does not exist: {self.repo_path}") if not os.path.isdir(self.repo_path): - raise ValueError(f"Primary repository path is not a directory: {self.repo_path}") - logger.info(f" ├─ ✓ Primary path valid: {self.repo_path}") - - # Validate additional paths if configured - if self.additional_source_paths: - logger.info(f" ├─ Validating {len(self.additional_source_paths)} additional path(s)...") - for i, path in enumerate(self.additional_source_paths, start=1): - logger.debug(f" │ └─ Checking additional path #{i}: {path}") - if not os.path.exists(path): - raise ValueError(f"Additional source path #{i} does not exist: {path}") - if not os.path.isdir(path): - raise ValueError(f"Additional source path #{i} is not a directory: {path}") - - # Check for read access - if not os.access(path, os.R_OK): - raise OSError(f"Additional source path #{i} is not readable: {path}") - logger.info(f" │ ├─ ✓ Additional path #{i} valid: {path}") - logger.info(f" └─ ✓ All {len(self.additional_source_paths)} additional path(s) validated") - else: - logger.debug(f" └─ No additional paths to validate (single-path mode)") - - def is_multi_path_mode(self) -> bool: - """ - Check if configuration is in multi-path mode. - - Returns: - True if additional_source_paths is set and non-empty, False otherwise - """ - import logging - logger = logging.getLogger(__name__) - - result = bool(self.additional_source_paths) - if result: - logger.debug(f"🔍 Multi-path mode: ENABLED ({len(self.additional_source_paths)} additional paths)") - else: - logger.debug("🔍 Multi-path mode: DISABLED (single-path)") - return result + raise ValueError(f"Repository path is not a directory: {self.repo_path}") + logger.info(f" └─ ✓ Repository path valid: {self.repo_path}") def get_prompt_addition(self) -> str: """Generate prompt additions based on agent instructions.""" @@ -416,8 +351,7 @@ def from_cli( main_max_token_field: str = "max_tokens", fallback_max_token_field: str = "max_tokens", agent_instructions: Optional[Dict[str, Any]] = None, - diagrams_dir: str = None, - additional_source_paths: Optional[List[str]] = None + diagrams_dir: str = None ) -> 'Config': """ Create configuration for CLI context. @@ -454,7 +388,6 @@ def from_cli( fallback_max_token_field: Parameter name for fallback model max tokens agent_instructions: Custom agent instructions dict diagrams_dir: Optional separate directory for Mermaid diagrams - additional_source_paths: Additional directories to analyze (multi-path mode) Returns: Config instance @@ -660,12 +593,11 @@ def from_cli( main_max_token_field=main_max_token_field, fallback_max_token_field=fallback_max_token_field, agent_instructions=agent_instructions, - diagrams_dir=diagrams_dir, - additional_source_paths=additional_source_paths + diagrams_dir=diagrams_dir ) - # Validate all source paths exist and are accessible - config.validate_source_paths() + # Validate the repository path exists and is a directory + config.validate_repo_path() return config @@ -718,13 +650,6 @@ def from_config_manager( if not config_obj.fallback_model: raise ValueError("fallback_model is not configured") - # Extract additional_source_paths from agent_instructions if present - additional_paths = None - if config_obj.agent_instructions: - agent_dict = config_obj.agent_instructions.to_dict() if hasattr(config_obj.agent_instructions, 'to_dict') else config_obj.agent_instructions - if isinstance(agent_dict, dict): - additional_paths = agent_dict.get('additional_source_paths') - # Use from_cli with all configuration from ConfigManager return cls.from_cli( repo_path=repo_path, @@ -757,6 +682,5 @@ def from_config_manager( main_max_token_field=config_obj.main_max_token_field, fallback_max_token_field=config_obj.fallback_max_token_field, agent_instructions=config_obj.agent_instructions.to_dict() if config_obj.agent_instructions else None, - diagrams_dir=None, - additional_source_paths=additional_paths + diagrams_dir=None ) diff --git a/docs/README.md b/docs/README.md index 849abb09..4665d75b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -64,11 +64,6 @@ Technical reference documentation generated directly from source code analysis: **Config Core** (`codewiki/src/config.py`) — Shared runtime configuration - [Config Core Overview](./reference/architecture/config-core/config-core.md) -**Test Multi Path Core** (`test-multi-path`) — Multi-source-path analysis validation -- [Test Multi Path Core Overview](./reference/architecture/test-multi-path-core/test-multi-path-core.md) -- [Test Fixtures](./reference/architecture/test-multi-path-core/test_fixtures.md) -- [Test Runners](./reference/architecture/test-multi-path-core/test_runners.md) - **Test Clustering Core** (`test_clustering`) — Module clustering validation - [Test Clustering Core Overview](./reference/architecture/test-clustering-core/test-clustering-core.md) - [Live Clustering Tests](./reference/architecture/test-clustering-core/live_clustering_tests.md) diff --git a/docs/diagrams/architecture/README-3.mmd b/docs/diagrams/architecture/README-3.mmd index 63c9b7e3..debc1ef8 100644 --- a/docs/diagrams/architecture/README-3.mmd +++ b/docs/diagrams/architecture/README-3.mmd @@ -7,8 +7,5 @@ flowchart LR Backend --> Agents["Agent Orchestration"] Backend --> Docs["Documentation Generation"] - MultiPathTests["Test Multi Path Core"] --> Config - MultiPathTests --> Analysis - ClusteringTests["Test Clustering Core"] --> Config ClusteringTests --> Backend diff --git a/docs/diagrams/architecture/test-multi-path-core-2.mmd b/docs/diagrams/architecture/test-multi-path-core-2.mmd deleted file mode 100644 index 6e475616..00000000 --- a/docs/diagrams/architecture/test-multi-path-core-2.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart LR - TMC["Test Multi Path Core"] -->|"configures"| ConfigCore["Config Core"] - TMC -->|"invokes"| BackendCore["Backend Core
DependencyGraphBuilder"] - BackendCore -->|"produces"| Graph["Component
Dependency Graph"] - TMC -->|"asserts on"| Graph diff --git a/docs/diagrams/architecture/test-multi-path-core.mmd b/docs/diagrams/architecture/test-multi-path-core.mmd deleted file mode 100644 index 36bc601c..00000000 --- a/docs/diagrams/architecture/test-multi-path-core.mmd +++ /dev/null @@ -1,29 +0,0 @@ -flowchart TD - subgraph fixtures_area["Test Fixtures"] - MainSvc["MainService
main/service.py"] - APICtrl["APIController
main/controller.py"] - PluginIface["PluginInterface
external/plugin.py"] - DataPlug["DataPlugin
external/plugin.py"] - end - - subgraph runners_area["Test Runners"] - IntTest["IntegrationTestRunner
integration_test.py"] - IntResults["TestResults
(integration_test)"] - MultiPathColors["Colors
test_multi_path.py"] - MultiPathResults["TestResults
(test_multi_path)"] - end - - IntTest -->|"builds"| DGB["DependencyGraphBuilder"] - MultiPathResults -.->|"used by"| MultiPathFuncs["test_* functions"] - MultiPathFuncs -->|"builds"| DGB - - DGB -->|"analyzes"| MainSvc - DGB -->|"analyzes"| APICtrl - DGB -->|"analyzes"| PluginIface - DGB -->|"analyzes"| DataPlug - - IntTest -->|"records"| IntResults - APICtrl -->|"depends on"| MainSvc - DataPlug -->|"implements"| PluginIface - - DGB -->|"reads"| Cfg["Config"] diff --git a/docs/diagrams/architecture/test_fixtures-2.mmd b/docs/diagrams/architecture/test_fixtures-2.mmd deleted file mode 100644 index a79c26b9..00000000 --- a/docs/diagrams/architecture/test_fixtures-2.mmd +++ /dev/null @@ -1,18 +0,0 @@ -flowchart LR - subgraph MainPkg["main package"] - Controller["APIController"] - Service["MainService"] - end - subgraph DepsPkg["deps package (external helper)"] - Validate["validate_input()"] - Process["process_data()"] - end - subgraph PluginPkg["external package"] - Base["PluginInterface"] - Plugin["DataPlugin"] - end - - Controller -->|"creates & delegates"| Service - Service -->|"calls"| Validate - Service -->|"calls"| Process - Plugin -->|"extends"| Base diff --git a/docs/diagrams/architecture/test_fixtures-3.mmd b/docs/diagrams/architecture/test_fixtures-3.mmd deleted file mode 100644 index fd771587..00000000 --- a/docs/diagrams/architecture/test_fixtures-3.mmd +++ /dev/null @@ -1,17 +0,0 @@ -sequenceDiagram - participant Client - participant Controller as "APIController" - participant Service as "MainService" - participant Helper as "deps.helper" - - Client->>Controller: handle_request("/process", data) - Controller->>Controller: request_count += 1 - Controller->>Service: process_request(data) - alt service not active - Service-->>Controller: raise RuntimeError - else service active - Service->>Helper: process_data(data) - Helper-->>Service: processed - Service-->>Controller: status success response - end - Controller-->>Client: response dict diff --git a/docs/diagrams/architecture/test_fixtures-4.mmd b/docs/diagrams/architecture/test_fixtures-4.mmd deleted file mode 100644 index 038b2e01..00000000 --- a/docs/diagrams/architecture/test_fixtures-4.mmd +++ /dev/null @@ -1,6 +0,0 @@ -flowchart TD - Create["Create DataPlugin(name)"] --> Init["initialize()"] - Init -->|"sets initialized = True"| Ready["Plugin Ready"] - Ready --> Exec["execute(context)"] - Exec -->|"initialized is True"| Result["Return processed result"] - Exec -->|"initialized is False"| Error["Raise RuntimeError"] diff --git a/docs/diagrams/architecture/test_fixtures.mmd b/docs/diagrams/architecture/test_fixtures.mmd deleted file mode 100644 index 8ae058b1..00000000 --- a/docs/diagrams/architecture/test_fixtures.mmd +++ /dev/null @@ -1,26 +0,0 @@ -classDiagram - class PluginInterface { - +initialize() bool - +execute(context: dict) dict - } - class DataPlugin { - +name: str - +initialized: bool - +initialize() bool - +execute(context: dict) dict - } - class MainService { - +name: str - +active: bool - +start(config: dict) bool - +process_request(data: dict) dict - +stop() - } - class APIController { - +service: MainService - +request_count: int - +handle_request(endpoint: str, data: dict) dict - } - - PluginInterface <|-- DataPlugin - APIController --> MainService : owns diff --git a/docs/diagrams/architecture/test_runners-2.mmd b/docs/diagrams/architecture/test_runners-2.mmd deleted file mode 100644 index abf82a28..00000000 --- a/docs/diagrams/architecture/test_runners-2.mmd +++ /dev/null @@ -1,42 +0,0 @@ -sequenceDiagram - participant Main as main() - participant ITR as IntegrationTestRunner - participant FS as Temp Filesystem - participant CFG as Config - participant DGB as DependencyGraphBuilder - participant TR as TestResults - - Main->>ITR: run() - ITR->>FS: setup_test_environment() - FS-->>ITR: main/, deps/, vendor/ created - ITR->>TR: add_test("Test directories created", True) - - ITR->>CFG: create_config() - CFG-->>ITR: Config(repo_path, additional_source_paths) - ITR->>TR: add_test("Config created with 3 source paths", True) - - ITR->>FS: validate_paths() - ITR->>TR: add_test("All paths validated successfully", ...) - - ITR->>DGB: DependencyGraphBuilder(config) - ITR->>DGB: build_dependency_graph() - DGB-->>ITR: (components, leaf_nodes) - ITR->>TR: add_test("Multi-path mode detected", ...) - - ITR->>ITR: verify_namespaces() - ITR->>TR: add_test("All expected namespaces present", ...) - ITR->>TR: add_test("Component counts match expectations", ...) - - ITR->>ITR: verify_cross_path_dependencies() - ITR->>TR: add_test("Cross-path dependencies detected", ...) - - ITR->>ITR: verify_no_warnings() - ITR->>TR: add_test("No warnings generated", ...) - - ITR->>ITR: verify_file_counts() - ITR->>TR: add_test("Component counts match expectations", ...) - - ITR->>ITR: print_detailed_output() - ITR->>TR: print_summary() - ITR->>FS: cleanup() [finally] - ITR-->>Main: exit code (0, 1, or 2) diff --git a/docs/diagrams/architecture/test_runners-3.mmd b/docs/diagrams/architecture/test_runners-3.mmd deleted file mode 100644 index f8e1fcab..00000000 --- a/docs/diagrams/architecture/test_runners-3.mmd +++ /dev/null @@ -1,24 +0,0 @@ -flowchart TD - Start["run_all_tests()"] --> T1["test_single_path()"] - Start --> T2["test_multiple_paths()"] - Start --> T3["test_component_namespacing()"] - Start --> T4["test_cross_path_dependencies()"] - Start --> T5["test_invalid_path_handling()"] - Start --> T6["test_empty_additional_paths()"] - Start --> T7["test_relative_vs_absolute_paths()"] - - T1 -->|"uses"| Cfg["create_test_config()"] - T2 -->|"uses"| Cfg - T3 -->|"uses"| Cfg - T4 -->|"uses"| Cfg - T5 -->|"uses"| Cfg - T6 -->|"uses"| Cfg - T7 -->|"uses"| Cfg - - Cfg -->|"builds"| ConfigObj["Config"] - ConfigObj -->|"passed to"| DGB["DependencyGraphBuilder"] - DGB -->|"build_dependency_graph()"| Components["components, leaf_nodes"] - - T1 & T2 & T3 & T4 & T5 & T6 & T7 -->|"boolean result"| Results["TestResults.add_test(name, passed)"] - Results --> Summary["TestResults.print_summary()"] - Summary --> Exit["process exit code"] diff --git a/docs/diagrams/architecture/test_runners.mmd b/docs/diagrams/architecture/test_runners.mmd deleted file mode 100644 index 765bf1d1..00000000 --- a/docs/diagrams/architecture/test_runners.mmd +++ /dev/null @@ -1,30 +0,0 @@ -flowchart TD - subgraph TestRunners["Test Runners"] - ITR["IntegrationTestRunner"] - TR1["TestResults (integration_test.py)"] - COL["Colors"] - TR2["TestResults (test_multi_path.py)"] - FUNCS["Function-based tests
(test_single_path, test_multiple_paths, ...)"] - end - - subgraph Fixtures["Test Fixtures (sibling module)"] - PLUGIN["DataPlugin / PluginInterface"] - CTRL["APIController"] - SVC["MainService"] - end - - subgraph Pipeline["Dependency Analysis Pipeline (external)"] - CFG["Config"] - DGB["DependencyGraphBuilder"] - end - - ITR -->|"creates temp files & drives"| CFG - ITR --> TR1 - ITR -->|"instantiates"| DGB - DGB -->|"produces Node graph"| ITR - - FUNCS -->|"reads fixture files from"| Fixtures - FUNCS -->|"builds"| CFG - FUNCS -->|"instantiates"| DGB - FUNCS -->|"records outcomes in"| TR2 - FUNCS -->|"prints via"| COL diff --git a/docs/reference/architecture/README.md b/docs/reference/architecture/README.md index ff950685..829fdc4a 100644 --- a/docs/reference/architecture/README.md +++ b/docs/reference/architecture/README.md @@ -70,7 +70,6 @@ CodeWiki/ │ ├── be/ Backend analysis, orchestration, and documentation engine │ ├── fe/ FastAPI web application and background job processing │ └── config.py Shared runtime configuration -├── test-multi-path/ Multi-source-path dependency analysis fixtures and tests └── test_clustering/ Module clustering diagnostic and validation tests ``` @@ -82,7 +81,6 @@ CodeWiki/ | Backend Core | `codewiki/src/be` | Implements repository analysis, dependency graph construction, module clustering, agent orchestration, and Markdown generation. | [Backend Core](backend-core.md) | | Frontend Core | `codewiki/src/fe` | Provides the web-facing application for repository submission, asynchronous processing, status tracking, caching, and documentation delivery. | [Frontend Core](frontend-core.md) | | Config Core | `codewiki/src/config.py` | Defines the shared `Config` runtime object for paths, LLM providers, generation limits, source roots, and agent instructions. | [Config Core](config-core.md) | -| Test Multi Path Core | `test-multi-path` | Validates analysis across a primary repository and additional source directories. | [Test Multi Path Core](test-multi-path-core.md) | | Test Clustering Core | `test_clustering` | Validates live and offline module-clustering behavior, ID normalization, and response validation. | [Test Clustering Core](test-clustering-core.md) | ## Module Relationships @@ -97,9 +95,6 @@ flowchart LR Backend --> Agents["Agent Orchestration"] Backend --> Docs["Documentation Generation"] - MultiPathTests["Test Multi Path Core"] --> Config - MultiPathTests --> Analysis - ClusteringTests["Test Clustering Core"] --> Config ClusteringTests --> Backend ``` diff --git a/docs/reference/architecture/test-multi-path-core/test-multi-path-core.md b/docs/reference/architecture/test-multi-path-core/test-multi-path-core.md deleted file mode 100644 index e6eb3f8e..00000000 --- a/docs/reference/architecture/test-multi-path-core/test-multi-path-core.md +++ /dev/null @@ -1,85 +0,0 @@ -# Test Multi Path Core - -## Purpose - -The Test Multi Path Core module is a self-contained test fixture and validation suite for CodeWiki's **multi-path dependency analysis** capability. This capability allows the [Backend Core](backend-core.md) dependency analyzer to ingest source code from a primary repository root *plus* one or more `additional_source_paths` (configured via [Config Core](config-core.md)), and to build a single, correctly namespaced dependency graph across all of them. - -This module exists purely to exercise and verify that behavior end-to-end. It provides: - -1. A small, realistic **sample application** spread across multiple simulated source roots (`main/`, `deps/`/`vendor/`, `external/`) with intentional cross-path imports. -2. **Test runner scripts** that configure the dependency analyzer against these fixtures, execute the full analysis pipeline, and assert on the resulting component graph — checking namespace correctness, component counts, cross-path dependency resolution, and error handling for invalid paths. - -Because it drives the real `DependencyGraphBuilder` and `Config` components from [Backend Core](backend-core.md) and [Config Core](config-core.md), this module doubles as executable documentation of how multi-path analysis is expected to behave. - -## Architecture Overview - -The module is organized into two complementary areas: static **fixture code** that stands in for a multi-directory codebase, and **test runner scripts** that drive analysis against those fixtures and validate the outcome. - -```mermaid -flowchart TD - subgraph fixtures_area["Test Fixtures"] - MainSvc["MainService
main/service.py"] - APICtrl["APIController
main/controller.py"] - PluginIface["PluginInterface
external/plugin.py"] - DataPlug["DataPlugin
external/plugin.py"] - end - - subgraph runners_area["Test Runners"] - IntTest["IntegrationTestRunner
integration_test.py"] - IntResults["TestResults
(integration_test)"] - MultiPathColors["Colors
test_multi_path.py"] - MultiPathResults["TestResults
(test_multi_path)"] - end - - IntTest -->|"builds"| DGB["DependencyGraphBuilder"] - MultiPathResults -.->|"used by"| MultiPathFuncs["test_* functions"] - MultiPathFuncs -->|"builds"| DGB - - DGB -->|"analyzes"| MainSvc - DGB -->|"analyzes"| APICtrl - DGB -->|"analyzes"| PluginIface - DGB -->|"analyzes"| DataPlug - - IntTest -->|"records"| IntResults - APICtrl -->|"depends on"| MainSvc - DataPlug -->|"implements"| PluginIface - - DGB -->|"reads"| Cfg["Config"] -``` - -Key relationships: - -- **Fixture components** (`MainService`, `APIController`, `PluginInterface`, `DataPlugin`) are ordinary application code with no awareness of testing — they exist only to be *discovered and analyzed* by the dependency analyzer running against multiple source roots. -- **Runner components** (`IntegrationTestRunner`, both `TestResults` classes, `Colors`) orchestrate the analysis pipeline (`Config` → `DependencyGraphBuilder`) against the fixtures and assert on the output. -- Both runner scripts independently import `Config` from [Config Core](config-core.md) and `DependencyGraphBuilder` from [Backend Core](backend-core.md), meaning this module is a **consumer** of those modules rather than a dependency of them. - -## Sub-Modules - -### Test Fixtures - -Sample application code intentionally spread across separate directories (`main/`, `external/`, and referenced `deps/`/`vendor/` paths) to simulate a realistic multi-root repository. Includes a small service/controller pair with a cross-file dependency, and a third-party-style plugin interface with a concrete implementation. This code is not exercised directly by end users — it is the input data that the test runners feed into the dependency analyzer. - -See [Test Fixtures](test_fixtures.md) for details on `MainService`, `APIController`, `PluginInterface`, and `DataPlugin`. - -### Test Runners - -Two independent, executable test scripts that each configure `Config` and `DependencyGraphBuilder` against the fixtures and validate results: `integration_test.py` runs a full generated-fixture end-to-end scenario (creating its own temporary files for `main/`, `deps/`, and `vendor/`), while `test_multi_path.py` runs a suite of scenario-based checks (single path, multiple paths, namespacing, cross-path dependencies, invalid paths, empty paths, relative vs. absolute paths) against the on-disk fixtures in this module. - -See [Test Runners](test_runners.md) for details on `IntegrationTestRunner`, `TestResults`, and `Colors`. - -## How This Module Fits Into the System - -```mermaid -flowchart LR - TMC["Test Multi Path Core"] -->|"configures"| ConfigCore["Config Core"] - TMC -->|"invokes"| BackendCore["Backend Core
DependencyGraphBuilder"] - BackendCore -->|"produces"| Graph["Component
Dependency Graph"] - TMC -->|"asserts on"| Graph -``` - -Test Multi Path Core does not provide functionality consumed by other modules in the system — it is a standalone validation harness. It depends on: - -- **[Config Core](config-core.md)** — for constructing `Config` instances with `additional_source_paths` set, which is the mechanism that enables multi-path mode. -- **[Backend Core](backend-core.md)** — specifically the dependency-analysis pipeline (`DependencyGraphBuilder`), which is the system under test. - -Its outputs are console reports and pass/fail exit codes rather than artifacts consumed elsewhere in the platform. diff --git a/docs/reference/architecture/test-multi-path-core/test_fixtures.md b/docs/reference/architecture/test-multi-path-core/test_fixtures.md deleted file mode 100644 index e650c8fe..00000000 --- a/docs/reference/architecture/test-multi-path-core/test_fixtures.md +++ /dev/null @@ -1,135 +0,0 @@ -# Test Fixtures - -The Test Fixtures module provides a small, self-contained set of sample application components used as **fixture code** for exercising the CodeWiki dependency-analysis and documentation-generation pipeline. It simulates a realistic multi-path Python project consisting of an API controller, a business service, and a third-party style plugin system. These fixtures are not part of CodeWiki's production runtime — they exist so that other CodeWiki subsystems (such as the [Backend Core](backend-core.md) dependency analyzers) have representative, cross-module code to parse, resolve, and document during testing. - -## Purpose and Scope - -This module is a child of the [Test Multi Path Core](test-multi-path-core.md) module. It contains the "application" half of the test fixture set: - -- A **plugin interface and implementation** (`external/plugin.py`) that mimics a third-party package with an abstract base class and a concrete subclass. -- A **main service** (`main/service.py`) that represents core business logic and depends on an external helper package (`deps.helper`). -- An **API controller** (`main/controller.py`) that sits in front of the service and routes simple HTTP-style endpoints to service methods. - -The sibling module, [Test Runners](test_runners.md), contains the test harnesses (`IntegrationTestRunner`, `TestResults`, `Colors`) that exercise these fixtures end-to-end. Together, the two child modules make up the full [Test Multi Path Core](test-multi-path-core.md) fixture project. - -## Core Components - -| Component | File | Responsibility | -|---|---|---| -| `PluginInterface` | `test-multi-path/external/plugin.py` | Abstract base class defining the plugin contract (`initialize`, `execute`). | -| `DataPlugin` | `test-multi-path/external/plugin.py` | Concrete plugin implementation that processes a `data` payload from a context dictionary. | -| `MainService` | `test-multi-path/main/service.py` | Core service exposing lifecycle (`start`/`stop`) and request-processing behavior; delegates data transformation to an external `deps.helper` module. | -| `APIController` | `test-multi-path/main/controller.py` | Thin routing layer that owns a `MainService` instance and dispatches requests based on endpoint string. | - -## Architecture - -The fixture code models a typical layered application: a controller delegates to a service, while a separate plugin subsystem demonstrates cross-package inheritance that the dependency analyzer must resolve correctly (base class defined in `external/plugin.py`, subclass in the same file, but conceptually representing a "third-party" dependency). - -```mermaid -classDiagram - class PluginInterface { - +initialize() bool - +execute(context: dict) dict - } - class DataPlugin { - +name: str - +initialized: bool - +initialize() bool - +execute(context: dict) dict - } - class MainService { - +name: str - +active: bool - +start(config: dict) bool - +process_request(data: dict) dict - +stop() - } - class APIController { - +service: MainService - +request_count: int - +handle_request(endpoint: str, data: dict) dict - } - - PluginInterface <|-- DataPlugin - APIController --> MainService : owns -``` - -## Component Relationships and Dependencies - -- `APIController` **instantiates and owns** a `MainService` object in its constructor, coupling the routing layer directly to a specific service instance. -- `MainService` depends on functions (`validate_input`, `process_data`) imported from an external `deps.helper` module. This module is outside the documented core components of this fixture set but represents the kind of cross-package call that the [Backend Core](backend-core.md) dependency-analysis subsystem is designed to trace. -- `DataPlugin` **inherits from** `PluginInterface`, overriding both `initialize()` and `execute()`. This inheritance relationship is a deliberate fixture pattern for validating that call-graph and class-hierarchy analyzers correctly link subclass methods back to their abstract base declarations. - -```mermaid -flowchart LR - subgraph MainPkg["main package"] - Controller["APIController"] - Service["MainService"] - end - subgraph DepsPkg["deps package (external helper)"] - Validate["validate_input()"] - Process["process_data()"] - end - subgraph PluginPkg["external package"] - Base["PluginInterface"] - Plugin["DataPlugin"] - end - - Controller -->|"creates & delegates"| Service - Service -->|"calls"| Validate - Service -->|"calls"| Process - Plugin -->|"extends"| Base -``` - -## Request Processing Flow - -The controller-to-service interaction demonstrates a simple, single-hop request pipeline used as a fixture for tracing call graphs across file boundaries. - -```mermaid -sequenceDiagram - participant Client - participant Controller as "APIController" - participant Service as "MainService" - participant Helper as "deps.helper" - - Client->>Controller: handle_request("/process", data) - Controller->>Controller: request_count += 1 - Controller->>Service: process_request(data) - alt service not active - Service-->>Controller: raise RuntimeError - else service active - Service->>Helper: process_data(data) - Helper-->>Service: processed - Service-->>Controller: status success response - end - Controller-->>Client: response dict -``` - -Key behavioral notes captured by this fixture: - -- `APIController.handle_request` supports two endpoints — `/process` (delegates to `MainService.process_request`) and `/health` (returns a status payload including the running `request_count`) — and returns an error payload for anything else. -- `MainService.process_request` raises a `RuntimeError` if the service has not been started via `start()`, exercising error-path analysis. -- `MainService.start` validates its `config` argument via `validate_input` before flipping the `active` flag to `True`. - -## Plugin Lifecycle Flow - -`DataPlugin` models a two-phase lifecycle — initialization followed by execution — common in plugin-based architectures. This is used as a fixture to validate that analyzers correctly track state (`initialized`) set in one method and checked in another. - -```mermaid -flowchart TD - Create["Create DataPlugin(name)"] --> Init["initialize()"] - Init -->|"sets initialized = True"| Ready["Plugin Ready"] - Ready --> Exec["execute(context)"] - Exec -->|"initialized is True"| Result["Return processed result"] - Exec -->|"initialized is False"| Error["Raise RuntimeError"] -``` - -## Role in the Test Multi-Path Fixture Set - -This module exists purely to provide realistic, multi-file, multi-package Python source for CodeWiki's analysis tooling to consume. It does not implement production functionality. Consumers of this fixture set typically: - -1. Point the [Backend Core](backend-core.md) dependency-analysis pipeline at the `test-multi-path` directory tree. -2. Verify that the resulting dependency graph correctly links `APIController` to `MainService` to `deps.helper`, and `DataPlugin` to `PluginInterface`. -3. Use the [Test Runners](test_runners.md) module's `IntegrationTestRunner` to validate the end-to-end analysis and documentation output against these fixtures. - -For information on how the overall fixture project is organized and how it relates to the top-level test harness, see [Test Multi Path Core](test-multi-path-core.md). diff --git a/docs/reference/architecture/test-multi-path-core/test_runners.md b/docs/reference/architecture/test-multi-path-core/test_runners.md deleted file mode 100644 index 76d18180..00000000 --- a/docs/reference/architecture/test-multi-path-core/test_runners.md +++ /dev/null @@ -1,235 +0,0 @@ -# Test Runners - -## Introduction - -The Test Runners module contains the executable test drivers used to validate CodeWiki's **multi-path source analysis** feature — the capability that allows the dependency analyzer to ingest code from a main repository plus one or more additional source directories (e.g. `deps/`, `vendor/`, `external/`) while keeping component identities correctly namespaced and dependency edges correctly resolved. - -This module provides two independent, script-style test drivers: - -- **`IntegrationTestRunner`** (in `integration_test.py`) — a self-contained, end-to-end integration test that builds its own temporary fixture files on disk, runs the full analysis pipeline, and asserts on the resulting component graph. -- **A function-based test suite** (in `test_multi_path.py`) built around the `Colors` and `TestResults` helper classes — a collection of discrete test functions that exercise multi-path analysis against pre-existing fixture directories (see the sibling [Test Fixtures](../test_fixtures/test_fixtures.md) module). - -Both drivers ultimately invoke the same production pipeline components — `Config` and `DependencyGraphBuilder` — that live outside this module, and both produce a human-readable pass/fail summary plus a process exit code suitable for CI. - -## Purpose and Scope - -The Test Runners module is a **verification harness**, not a production component. Its responsibilities are: - -1. Construct realistic multi-path repository layouts (either programmatically, in the case of `IntegrationTestRunner`, or by reading pre-built fixtures, in the case of `test_multi_path.py`). -2. Drive the dependency-analysis pipeline (`Config` → `DependencyGraphBuilder`) against those layouts. -3. Assert that: - - Components from every source path are discovered. - - Component IDs are namespaced by their source path (no ID collisions). - - Cross-path dependencies are captured where possible. - - No unexpected files/paths are missing or invalid. -4. Aggregate individual assertion results into a single summary report and translate the outcome into a process exit code (`0` = success, non-zero = failure). - -## Architecture Overview - -```mermaid -flowchart TD - subgraph TestRunners["Test Runners"] - ITR["IntegrationTestRunner"] - TR1["TestResults (integration_test.py)"] - COL["Colors"] - TR2["TestResults (test_multi_path.py)"] - FUNCS["Function-based tests
(test_single_path, test_multiple_paths, ...)"] - end - - subgraph Fixtures["Test Fixtures (sibling module)"] - PLUGIN["DataPlugin / PluginInterface"] - CTRL["APIController"] - SVC["MainService"] - end - - subgraph Pipeline["Dependency Analysis Pipeline (external)"] - CFG["Config"] - DGB["DependencyGraphBuilder"] - end - - ITR -->|"creates temp files & drives"| CFG - ITR --> TR1 - ITR -->|"instantiates"| DGB - DGB -->|"produces Node graph"| ITR - - FUNCS -->|"reads fixture files from"| Fixtures - FUNCS -->|"builds"| CFG - FUNCS -->|"instantiates"| DGB - FUNCS -->|"records outcomes in"| TR2 - FUNCS -->|"prints via"| COL -``` - -The two drivers are architecturally parallel but do not share code: each defines its own `TestResults` class with a slightly different API, reflecting their independent evolution as standalone scripts. Both, however, converge on the same two external dependencies: - -- **`Config`** — see the [Config Core](config-core.md) module for configuration fields such as `repo_path`, `additional_source_paths`, and `is_multi_path_mode()`. -- **`DependencyGraphBuilder`** — see the [Backend Core](backend-core.md) module for how the multi-path component graph is actually built. - -## Core Components - -### IntegrationTestRunner - -`IntegrationTestRunner` is a stateful orchestrator class that owns the entire lifecycle of a single, self-contained end-to-end test run. Unlike the function-based suite, it does not rely on pre-existing fixture files — it programmatically creates a temporary directory tree with three source roots (`main/`, `deps/`, `vendor/`) and hand-written Python files exhibiting realistic intra- and cross-path imports. - -**Key responsibilities (mapped to instance methods):** - -| Method | Responsibility | -|---|---| -| `setup_test_environment()` | Creates a temp directory with `main/`, `deps/`, `vendor/` subfolders and populates each with sample `.py` files (`_create_main_files`, `_create_deps_files`, `_create_vendor_files`). | -| `create_config()` | Builds a `Config` instance with `repo_path` set to `main/` and `additional_source_paths` set to `[deps/, vendor/]`. | -| `validate_paths()` | Confirms the root and all additional paths exist on disk. | -| `execute_dependency_parser()` | Instantiates `DependencyGraphBuilder(config)`, checks `config.is_multi_path_mode()`, and calls `build_dependency_graph()` to obtain `(components, leaf_nodes)`. | -| `verify_namespaces()` / `_verify_namespace_counts()` | Groups discovered component IDs by top-level namespace (`main`, `deps`, `vendor`) and checks expected counts (7, 5, 2 respectively). | -| `verify_cross_path_dependencies()` | Inspects each component's `dependencies` attribute for edges that cross namespace boundaries. | -| `verify_no_warnings()` | Checks whether the builder exposes a `warnings` attribute and, if so, that it is empty. | -| `verify_file_counts()` | Confirms the total component count across all namespaces matches the expected total (14). | -| `print_detailed_output()` | Prints all discovered component IDs, cross-namespace dependencies, and per-namespace counts for manual inspection. | -| `cleanup()` | Removes the temporary test directory (`shutil.rmtree`), always executed in the `finally` block of `run()`. | -| `run()` | Orchestrates the full sequence above in order and returns an integer exit code (`0` success, `1` assertion failure, `2` unhandled exception). | - -**Instance state** tracked across the run: `test_dir`, `main_path`, `deps_path`, `vendor_path`, `config`, `builder`, `components`, `leaf_nodes`, and a `results: TestResults` accumulator. - -### TestResults (integration_test.py) - -A lightweight accumulator used exclusively by `IntegrationTestRunner`. Each recorded entry is a dictionary with `name`, `passed`, and an optional `details` string (used both for failure diagnostics and non-fatal warnings). - -- `add_test(name, passed, details="")` — appends a result entry. -- `print_summary()` — prints total/passed/failed counts, lists failed assertions with their details, lists any "warning" entries (passed assertions that still carry a `details` message), and prints a final `✅ INTEGRATION TEST PASSED` / `❌ INTEGRATION TEST FAILED` banner. -- `all_passed()` — returns `True` only if every recorded entry passed; used by `IntegrationTestRunner.run()` to compute the process exit code. - -### Colors - -A minimal ANSI escape-code namespace used by the function-based test suite in `test_multi_path.py` for colorized console output. It defines class-level string constants: - -- `GREEN`, `RED`, `YELLOW`, `BLUE`, `BOLD`, `END` - -These are consumed by the module-level helper functions `print_header`, `print_success`, `print_error`, and `print_warning` to produce readable, color-coded terminal output while the test suite runs. - -### TestResults (test_multi_path.py) - -A separate, simpler accumulator (same class name as above, but a distinct implementation local to `test_multi_path.py`). It stores results as a `Dict[str, bool]` keyed by test name rather than a list of dictionaries with details. - -- `add_test(name, passed)` — records a boolean outcome for a named test. -- `print_summary()` — prints a formatted header, iterates all results printing `print_success`/`print_error` per test (using `Colors`), prints a `"passed/total"` tally, and returns `0` if all tests passed or `1` otherwise — directly usable as a process exit code. - -> **Note:** The two `TestResults` classes are intentionally independent implementations with different constructors and method signatures (`add_test(name, passed, details="")` returning nothing and printing a rich report, versus `add_test(name, passed)` with `print_summary()` returning an `int`). They are not meant to be interchanged; each is scoped to its own script. - -## Data Flow: IntegrationTestRunner Execution - -```mermaid -sequenceDiagram - participant Main as main() - participant ITR as IntegrationTestRunner - participant FS as Temp Filesystem - participant CFG as Config - participant DGB as DependencyGraphBuilder - participant TR as TestResults - - Main->>ITR: run() - ITR->>FS: setup_test_environment() - FS-->>ITR: main/, deps/, vendor/ created - ITR->>TR: add_test("Test directories created", True) - - ITR->>CFG: create_config() - CFG-->>ITR: Config(repo_path, additional_source_paths) - ITR->>TR: add_test("Config created with 3 source paths", True) - - ITR->>FS: validate_paths() - ITR->>TR: add_test("All paths validated successfully", ...) - - ITR->>DGB: DependencyGraphBuilder(config) - ITR->>DGB: build_dependency_graph() - DGB-->>ITR: (components, leaf_nodes) - ITR->>TR: add_test("Multi-path mode detected", ...) - - ITR->>ITR: verify_namespaces() - ITR->>TR: add_test("All expected namespaces present", ...) - ITR->>TR: add_test("Component counts match expectations", ...) - - ITR->>ITR: verify_cross_path_dependencies() - ITR->>TR: add_test("Cross-path dependencies detected", ...) - - ITR->>ITR: verify_no_warnings() - ITR->>TR: add_test("No warnings generated", ...) - - ITR->>ITR: verify_file_counts() - ITR->>TR: add_test("Component counts match expectations", ...) - - ITR->>ITR: print_detailed_output() - ITR->>TR: print_summary() - ITR->>FS: cleanup() [finally] - ITR-->>Main: exit code (0, 1, or 2) -``` - -## Data Flow: Function-Based Multi-Path Test Suite - -The `test_multi_path.py` driver takes a different shape: rather than one stateful orchestrator, it defines a series of independent test functions, each returning a boolean, which are collected and executed by `run_all_tests()`. - -```mermaid -flowchart TD - Start["run_all_tests()"] --> T1["test_single_path()"] - Start --> T2["test_multiple_paths()"] - Start --> T3["test_component_namespacing()"] - Start --> T4["test_cross_path_dependencies()"] - Start --> T5["test_invalid_path_handling()"] - Start --> T6["test_empty_additional_paths()"] - Start --> T7["test_relative_vs_absolute_paths()"] - - T1 -->|"uses"| Cfg["create_test_config()"] - T2 -->|"uses"| Cfg - T3 -->|"uses"| Cfg - T4 -->|"uses"| Cfg - T5 -->|"uses"| Cfg - T6 -->|"uses"| Cfg - T7 -->|"uses"| Cfg - - Cfg -->|"builds"| ConfigObj["Config"] - ConfigObj -->|"passed to"| DGB["DependencyGraphBuilder"] - DGB -->|"build_dependency_graph()"| Components["components, leaf_nodes"] - - T1 & T2 & T3 & T4 & T5 & T6 & T7 -->|"boolean result"| Results["TestResults.add_test(name, passed)"] - Results --> Summary["TestResults.print_summary()"] - Summary --> Exit["process exit code"] -``` - -Each test function follows the same shape: - -1. Resolve fixture paths (`main/`, `deps/`, `external/`) relative to the script's own directory — these fixtures are the ones documented in the sibling [Test Fixtures](test_fixtures.md) module (`APIController`, `MainService`, `DataPlugin`, `PluginInterface`). -2. Build a `Config` via the local `create_test_config()` helper, optionally supplying `additional_source_paths`. -3. Instantiate `DependencyGraphBuilder(config)` and call `build_dependency_graph()`. -4. Convert the returned `Node` objects to plain dictionaries via `model_dump()` for easy field inspection (`file_path`, `depends_on`). -5. Assert path-specific expectations (e.g., that `service.py` and `controller.py` are present under `main/`, that component IDs are unique, that cross-path dependency edges exist, that invalid paths raise `ValueError`/`OSError` from `config.validate_source_paths()`). -6. Return `True`/`False`, printed via the `Colors`-driven `print_success`/`print_error` helpers. - -Notable individual tests: - -- **`test_single_path`** — backward-compatibility check that a single `repo_path` with no `additional_source_paths` still analyzes correctly. -- **`test_multiple_paths`** — verifies that components from `main/`, `deps/`, and `external/` are all discovered in one pass. -- **`test_component_namespacing`** — asserts no duplicate component IDs are produced across paths and warns if an ID doesn't visibly reflect its source path. -- **`test_cross_path_dependencies`** — inspects each component's `depends_on` set to detect edges crossing between `main` and `deps`. -- **`test_invalid_path_handling`** — expects `config.validate_source_paths()` to raise on a nonexistent additional path. -- **`test_empty_additional_paths`** — confirms an empty `additional_source_paths` list behaves like the single-path case. -- **`test_relative_vs_absolute_paths`** — exercises path resolution behavior for relative vs. absolute additional paths. - -## Relationship to Other Modules - -- **[Test Fixtures](test_fixtures.md)** — the sibling module under [Test Multi Path Core](test-multi-path-core.md) that defines the on-disk sample components (`MainService`, `APIController`, `DataPlugin`, `PluginInterface`) consumed by `test_multi_path.py`'s function-based suite. `IntegrationTestRunner` does **not** depend on these fixtures; it generates its own files at runtime instead. -- **[Config Core](config-core.md)** — supplies the `repo_path` / `additional_source_paths` / `is_multi_path_mode()` / `validate_source_paths()` surface that both test drivers configure and exercise. -- **[Backend Core](backend-core.md)** — provides `DependencyGraphBuilder`, the production component whose multi-path behavior is the actual subject under test in this module. - -## Usage - -Both drivers are designed to be run as standalone scripts and return a process-appropriate exit code: - -```bash -# Run the self-contained end-to-end integration test -python integration_test.py - -# Run the fixture-based multi-path test suite -python test_multi_path.py -``` - -`integration_test.py` exits `0` on full success, `1` if any assertion failed, and `2` if the run crashed with an unhandled exception (stack trace printed via `traceback.print_exc()`). `test_multi_path.py`'s `run_all_tests()` returns `0` only if every individual test function returned `True`, otherwise `1`. - -## Summary - -The Test Runners module is the executable validation layer for CodeWiki's multi-path analysis capability. `IntegrationTestRunner` provides a fully self-contained, disk-based end-to-end scenario with rich diagnostic output, while the `Colors`/`TestResults` pairing in `test_multi_path.py` powers a broader, fixture-driven suite covering backward compatibility, namespacing, cross-path dependencies, and error handling. Together they validate the same underlying `Config` → `DependencyGraphBuilder` pipeline from two complementary angles, without depending on each other's internals. diff --git a/examples/.multi_path_example.md b/examples/.multi_path_example.md deleted file mode 100644 index f6f554c8..00000000 --- a/examples/.multi_path_example.md +++ /dev/null @@ -1,42 +0,0 @@ - -Demonstrates multi-path repository analysis using `DependencyParser` and `RepoAnalyzer`, showing how to analyze multiple source directories simultaneously with namespace-aware component tracking. - -## Key Components - -| Function | Description | -|---|---| -| `example_single_path()` | Backward-compatible single-path parsing with include/exclude patterns | -| `example_multiple_paths()` | Multi-repo parsing with automatic namespace grouping and cross-namespace dependency detection | -| `example_repo_analyzer_multi_path()` | High-level `RepoAnalyzer` usage across multiple repos, returning summary and file tree | -| `example_component_details()` | Accessing per-namespace component metadata (type, file path, line range, dependencies) | -| `example_saving_output()` | Persisting the full multi-path dependency graph to a JSON file | - -## Usage Example - -```python -from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser - -# Multi-path analysis with namespacing -parser = DependencyParser( - ["/path/to/frontend", "/path/to/backend", "/path/to/shared"], - include_patterns=["*.py", "*.ts", "*.tsx"], - exclude_patterns=["*test*", "*spec*", "node_modules/*"] -) - -components = parser.parse_repository() - -# Components are prefixed by namespace (derived from directory name) -for comp_id, component in components.items(): - namespace = comp_id.split('.')[0] # e.g., "frontend", "backend" - print(f"{namespace}: {comp_id} -> {component.name}") - -# Save the full graph to disk -parser.save_dependency_graph("/tmp/multi_path_analysis.json") -``` - -## Notes - -- All example functions use **placeholder paths** — replace `/path/to/...` with real directories before running. -- Component IDs are automatically **namespaced** by directory name (e.g., `frontend.src.components.Button`) to avoid collisions across repos. -- Cross-namespace dependencies are preserved and visible in the parsed output. -- Source: [`examples/multi_path_example.py`](https://github.com/flamingo-stack/CodeWiki/blob/main/examples/multi_path_example.py) \ No newline at end of file diff --git a/examples/multi_path_example.py b/examples/multi_path_example.py deleted file mode 100644 index e0a39533..00000000 --- a/examples/multi_path_example.py +++ /dev/null @@ -1,225 +0,0 @@ -#!/usr/bin/env python3 -""" -Example: Multi-Path Repository Analysis - -This script demonstrates how to use DependencyParser and RepoAnalyzer -with multiple source directories. -""" - -import sys -import os - -# Add CodeWiki to path -sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) - -from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser -from codewiki.src.be.dependency_analyzer.analysis.repo_analyzer import RepoAnalyzer - - -def example_single_path(): - """Example: Single path (backward compatible).""" - print("=" * 60) - print("EXAMPLE 1: Single Path (Backward Compatible)") - print("=" * 60) - - # Works exactly as before - parser = DependencyParser( - "/path/to/my-project", - include_patterns=["*.py", "*.ts"], - exclude_patterns=["*test*"] - ) - - components = parser.parse_repository() - - print(f"Found {len(components)} components") - print("\nSample component IDs:") - for comp_id in list(components.keys())[:5]: - print(f" - {comp_id}") - - -def example_multiple_paths(): - """Example: Multiple paths with namespacing.""" - print("\n" + "=" * 60) - print("EXAMPLE 2: Multiple Paths with Namespacing") - print("=" * 60) - - # Define multiple repositories to analyze - repositories = [ - "/path/to/openframe-frontend", - "/path/to/ui-kit", - "/path/to/shared-libraries" - ] - - # Create parser with multiple paths - parser = DependencyParser( - repositories, - include_patterns=["*.py", "*.ts", "*.tsx"], - exclude_patterns=["*test*", "*spec*", "node_modules/*"] - ) - - # Parse all repositories - components = parser.parse_repository() - - print(f"\nTotal components: {len(components)}") - print(f"Total modules: {len(parser.modules)}") - - # Group by namespace - namespaces = {} - for comp_id, component in components.items(): - namespace = comp_id.split('.')[0] - if namespace not in namespaces: - namespaces[namespace] = [] - namespaces[namespace].append(comp_id) - - print("\nComponents by namespace:") - for namespace, comp_list in namespaces.items(): - print(f" {namespace}: {len(comp_list)} components") - - # Show sample namespaced component IDs - print("\nSample namespaced component IDs:") - for comp_id in list(components.keys())[:10]: - print(f" - {comp_id}") - - # Show cross-namespace dependencies - print("\nCross-namespace dependencies:") - for comp_id, component in list(components.items())[:50]: - comp_namespace = comp_id.split('.')[0] - cross_deps = [ - dep for dep in component.depends_on - if dep.split('.')[0] != comp_namespace - ] - if cross_deps: - print(f" {comp_id} depends on:") - for dep in cross_deps[:3]: # Show first 3 - print(f" → {dep}") - if len(cross_deps) > 3: - print(f" ... and {len(cross_deps) - 3} more") - break # Just show first example - - -def example_repo_analyzer_multi_path(): - """Example: RepoAnalyzer with multiple paths.""" - print("\n" + "=" * 60) - print("EXAMPLE 3: RepoAnalyzer with Multiple Paths") - print("=" * 60) - - repositories = [ - "/path/to/frontend", - "/path/to/backend", - "/path/to/shared" - ] - - analyzer = RepoAnalyzer( - include_patterns=["*.py", "*.ts"], - exclude_patterns=["*test*", "build/*", "dist/*"] - ) - - result = analyzer.analyze_repository_structure(repositories) - - # Display summary - summary = result["summary"] - print(f"\nSummary:") - print(f" Total files: {summary['total_files']}") - print(f" Total size: {summary['total_size_kb']:.2f} KB") - print(f" Repositories: {summary['repositories']}") - print(f" Namespaces: {', '.join(summary['namespaces'])}") - - # Show file tree structure - print("\nFile tree structure (top level):") - file_tree = result["file_tree"] - for child in file_tree.get("children", [])[:3]: - namespace = child.get("_namespace", "unknown") - num_children = len(child.get("children", [])) - print(f" 📁 {namespace}/ ({num_children} items)") - - -def example_component_details(): - """Example: Accessing component details in multi-path mode.""" - print("\n" + "=" * 60) - print("EXAMPLE 4: Component Details in Multi-Path Mode") - print("=" * 60) - - parser = DependencyParser([ - "/path/to/project-a", - "/path/to/project-b" - ]) - - components = parser.parse_repository() - - # Find components from each namespace - for namespace in ["project-a", "project-b"]: - namespace_components = { - comp_id: comp for comp_id, comp in components.items() - if comp_id.startswith(f"{namespace}.") - } - - if namespace_components: - print(f"\n{namespace} components:") - sample_id = list(namespace_components.keys())[0] - sample_comp = namespace_components[sample_id] - - print(f" Sample: {sample_id}") - print(f" Name: {sample_comp.name}") - print(f" Type: {sample_comp.component_type}") - print(f" File: {sample_comp.relative_path}") - print(f" Lines: {sample_comp.start_line}-{sample_comp.end_line}") - print(f" Dependencies: {len(sample_comp.depends_on)}") - - -def example_saving_output(): - """Example: Saving multi-path analysis to JSON.""" - print("\n" + "=" * 60) - print("EXAMPLE 5: Saving Multi-Path Analysis") - print("=" * 60) - - parser = DependencyParser([ - "/path/to/repo1", - "/path/to/repo2" - ]) - - components = parser.parse_repository() - - # Save to JSON file - output_path = "/tmp/multi_path_analysis.json" - parser.save_dependency_graph(output_path) - - print(f"\nSaved {len(components)} components to:") - print(f" {output_path}") - print("\nJSON structure includes:") - print(" - Namespaced component IDs") - print(" - Cross-namespace dependencies") - print(" - Original file paths preserved") - print(" - Complete component metadata") - - -def main(): - """Run all examples.""" - print("\n") - print("╔" + "=" * 58 + "╗") - print("║" + " " * 10 + "Multi-Path Repository Analysis Examples" + " " * 8 + "║") - print("╚" + "=" * 58 + "╝") - - try: - # Note: These examples use placeholder paths - # Replace with actual repository paths to run - - print("\nNOTE: Examples use placeholder paths.") - print("Replace with actual paths to execute.\n") - - # Uncomment to run examples: - # example_single_path() - # example_multiple_paths() - # example_repo_analyzer_multi_path() - # example_component_details() - # example_saving_output() - - print("\n✅ Examples ready to run (update paths first)") - - except Exception as e: - print(f"\n❌ Error: {e}") - import traceback - traceback.print_exc() - - -if __name__ == "__main__": - main() diff --git a/pyproject.toml b/pyproject.toml index 651408c4..2e6e8d78 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -42,8 +42,9 @@ dependencies = [ "litellm>=1.77.0", "pydantic>=2.11.7", "pydantic-settings>=2.10.1", - "pydantic-ai>=2.0,<3.0", + "pydantic-ai>=2.16,<3.0", "requests>=2.32.4", + "httpx>=0.27", "python-dotenv>=1.1.1", "rich>=14.1.0", "networkx>=3.5", diff --git a/requirements.txt b/requirements.txt index 993722f3..1145e9ff 100644 --- a/requirements.txt +++ b/requirements.txt @@ -105,8 +105,8 @@ pyasn1_modules==0.4.2 pycares==4.11.0 pycparser==2.23 pydantic==2.11.7 -pydantic-ai>=2.0,<3.0 -pydantic-ai-slim>=2.0,<3.0 +pydantic-ai>=2.16,<3.0 +pydantic-ai-slim>=2.16,<3.0 pydantic-evals>=1.37.0 pydantic-graph>=1.37.0 pydantic-settings==2.10.1 diff --git a/test-multi-path/.debug_parser.md b/test-multi-path/.debug_parser.md deleted file mode 100644 index 5c04d630..00000000 --- a/test-multi-path/.debug_parser.md +++ /dev/null @@ -1,55 +0,0 @@ - -Diagnostic script that verifies the `DependencyParser` can locate and parse Python source files across single and multi-path repository configurations. - -## Key Components - -| Element | Description | -|---|---| -| `parser_single` | `DependencyParser` instance scanning only the `main/` subdirectory | -| `parser_multi` | `DependencyParser` instance scanning both `main/` and `deps/` subdirectories | -| `components_single` / `components_multi` | Parsed component dictionaries printed for quick inspection | - -## Usage Example - -```bash -# Run from the repository root -python debug_parser.py -``` - -Expected output: - -```text -Main path: /path/to/tests/main -Deps path: /path/to/tests/deps -Main exists: True -Deps exists: True - -Files in main: - - example.py - -Files in deps: - - helper.py - -====================================================================== -Testing single path parser... -====================================================================== - -Single path found 3 components - - example.MyClass: class - -====================================================================== -Testing multi-path parser... -====================================================================== - -Multi-path found 7 components - - helper.util_fn: function -``` - -## Notes - -- Prepends `CODEWIKI_ROOT` to `sys.path` so `codewiki` is importable without installation. -- Globs only `*.py` files when listing directory contents — non-Python files are ignored. -- Prints at most **5** components for the single-path run and **10** for the multi-path run to keep output readable. -- Useful when adding new test fixtures under `main/` or `deps/` to confirm the parser picks them up before running the full test suite. - -**Source:** [`debug_parser.py`](https://github.com/flamingo-stack/CodeWiki/blob/main/debug_parser.py) \ No newline at end of file diff --git a/test-multi-path/.integration_test.md b/test-multi-path/.integration_test.md deleted file mode 100644 index 481e40df..00000000 --- a/test-multi-path/.integration_test.md +++ /dev/null @@ -1,39 +0,0 @@ - -Concise documentation for `integration_test.py`: - ---- - -An end-to-end integration test script that validates CodeWiki's multi-path dependency analysis pipeline by creating a temporary multi-directory project (`main/`, `deps/`, `vendor/`) with cross-path Python imports, then running it through the full `Config` → `DependencyGraphBuilder` pipeline and asserting on the resulting component graph and namespacing. - -## Key Components - -- **`TestResults`** — Accumulates named pass/fail test outcomes with optional detail messages; provides `add_test()`, `print_summary()` (formatted report with counts and warnings), and `all_passed()` for overall status. -- **`IntegrationTestRunner`** — Orchestrates the full test workflow via sequential steps: - - `setup_test_environment()` — Creates a temp directory with `main/`, `deps/`, `vendor/` subdirectories. - - `_create_main_files()`, `_create_deps_files()`, `_create_vendor_files()` — Populate each directory with sample Python modules exhibiting realistic cross-path imports (e.g., `main/service.py` imports from `deps.helper` and `vendor.logger`). - - `create_config()` — Builds a `Config` object using `main/` as the repo root and `deps/`, `vendor/` as `additional_source_paths`. - - `validate_paths()` — Confirms all configured source paths exist on disk. - - `execute_dependency_parser()` — Instantiates `DependencyGraphBuilder`, verifies multi-path mode detection, and runs `build_dependency_graph()` to produce `components` and `leaf_nodes`. - - `verify_namespaces()` — Groups discovered components by top-level namespace (derived from the dotted component ID prefix, e.g., `main`, `deps`, `vendor`) and prints a breakdown for manual/assertion-based inspection. - -## Usage Example - -```python -# Run as a standalone script -python integration_test.py - -# Or drive it programmatically -from integration_test import IntegrationTestRunner - -runner = IntegrationTestRunner() -runner.setup_test_environment() -runner.create_config() -runner.validate_paths() -runner.execute_dependency_parser() -runner.verify_namespaces() - -runner.results.print_summary() -assert runner.results.all_passed() -``` - -**Note:** This script is intended to be executed directly from the `tests/` (or similar) directory adjacent to the `codewiki` package, since it inserts `CODEWIKI_ROOT` (`Path(__file__).parent.parent`) into `sys.path` to import `codewiki.src.config.Config` and `codewiki.src.be.dependency_analyzer.DependencyGraphBuilder`. Temporary directories are created via `tempfile.mkdtemp` and should be cleaned up (via `shutil`) after test completion. \ No newline at end of file diff --git a/test-multi-path/.test_multi_path.md b/test-multi-path/.test_multi_path.md deleted file mode 100644 index c05f4c71..00000000 --- a/test-multi-path/.test_multi_path.md +++ /dev/null @@ -1,36 +0,0 @@ - -## Overview - -`test_multi_path.py` is a manual/integration test script for CodeWiki's multi-path analysis feature. It verifies that the `DependencyGraphBuilder` can correctly ingest, namespace, and cross-reference components discovered across multiple source directories (a primary repo path plus additional external paths) without ID collisions or missed dependencies. - -## Key Components - -- **`Colors`** — ANSI color code constants used to format terminal output (success, error, warning, headers). -- **`TestResults`** — Accumulator class that records pass/fail status per test via `add_test()` and prints a final tally with `print_summary()`, returning a process-friendly exit code (`0` = all passed, `1` = failures). -- **Print helpers** — `print_header()`, `print_success()`, `print_error()`, `print_warning()`, `print_info()` for consistent colored console output. -- **`create_test_config(repo_path, output_dir, additional_paths=None)`** — Builds a `Config` object pointing to test fixture directories (`main/`, `deps/`, `external/`), wiring in placeholder LLM model/API settings for the `DependencyGraphBuilder`. -- **Test functions** — Each exercises a specific scenario against `DependencyGraphBuilder.build_dependency_graph()`: - - `test_single_path()` — Confirms backward compatibility when only one repo path is supplied. - - `test_multiple_paths()` — Confirms components from `main/`, `deps/`, and `external/` are all discovered. - - `test_component_namespacing()` — Confirms component IDs are unique across paths and encode path context to avoid collisions. - - `test_cross_path_dependencies()` — Confirms dependency edges correctly link components across different source paths (e.g., `main/service.py` importing `deps/helper.py`). - -## Usage Example - -```bash -# Run the test suite directly from the tests directory -python3 test_multi_path.py -``` - -```python -# Programmatic invocation of an individual test -from test_multi_path import test_single_path, test_multiple_paths, TestResults - -results = TestResults() -results.add_test("single_path", test_single_path()) -results.add_test("multiple_paths", test_multiple_paths()) - -exit_code = results.print_summary() -``` - -Each test expects fixture directories (`main/`, `deps/`, `external/`) to exist alongside the script, containing sample Python files (e.g., `service.py`, `controller.py`, `helper.py`, `utils.py`, `plugin.py`) used to validate discovery and dependency resolution. \ No newline at end of file diff --git a/test-multi-path/INTEGRATION_TEST_RESULTS.md b/test-multi-path/INTEGRATION_TEST_RESULTS.md deleted file mode 100644 index 2681a264..00000000 --- a/test-multi-path/INTEGRATION_TEST_RESULTS.md +++ /dev/null @@ -1,185 +0,0 @@ -# Multi-Path Integration Test Results - -## Test Overview -Complete end-to-end integration test for CodeWiki's multi-path analysis feature. - -**Test File**: `/Users/michaelassraf/Documents/GitHub/CodeWiki/test-multi-path/integration_test.py` - -**Execution Date**: February 2, 2026 - ---- - -## Test Results: ✅ PASSED (9/9 assertions) - -### Test Stages - -#### 1. ✅ Test Environment Setup -- Created temporary test directory structure -- Generated 10 Python files across 3 namespaces: - - `main/` - 5 files (service, api, models, controller, utils) - - `deps/` - 3 files (helper, validator, cache) - - `vendor/` - 2 files (logger, metrics) -- Files include realistic cross-path imports - -#### 2. ✅ Configuration Creation -- Created `Config` with: - - Primary path: `main/` - - Additional paths: `deps/`, `vendor/` -- Verified `is_multi_path_mode()` returns `True` - -#### 3. ✅ Path Validation -- Validated all 3 source paths exist -- Confirmed paths are accessible - -#### 4. ✅ Dependency Graph Building -- Executed `DependencyGraphBuilder.build_dependency_graph()` -- Successfully parsed **14 components** (classes + functions) -- Identified **8 leaf nodes** - -#### 5. ✅ Namespace Verification -- Verified all 3 expected namespaces present: `main`, `deps`, `vendor` -- Component namespace format: `namespace.module.Component` -- Example: `main.service.MainService`, `deps.helper.format_data` - -#### 6. ✅ Component Count Verification -- **main**: 7 components (5 classes + 2 functions) ✓ -- **deps**: 5 components (1 class + 4 functions) ✓ -- **vendor**: 2 components (2 classes) ✓ -- **Total**: 14 components ✓ - -#### 7. ✅ Cross-Path Dependencies -- **Status**: No cross-path dependencies detected (expected behavior) -- **Reason**: AST parser currently parses import statements but does not resolve them to specific components across namespaces -- **Future Enhancement**: Cross-namespace dependency resolution can be added later - -#### 8. ✅ Warning Validation -- No parser warnings generated -- Clean execution - -#### 9. ✅ Final Component Count -- Expected: 14 components -- Actual: 14 components -- Match: ✓ - ---- - -## Key Findings - -### ✅ Multi-Path Mode Works Correctly -1. **Configuration**: `additional_source_paths` parameter accepted -2. **Detection**: `is_multi_path_mode()` correctly identifies multi-path configs -3. **Path Aggregation**: `all_source_paths` includes all configured paths -4. **Parsing**: `DependencyGraphBuilder` analyzes all paths -5. **Namespacing**: Components correctly namespaced by source directory - -### ✅ Namespace Format -- **Pattern**: `..` -- **Examples**: - ``` - main.service.MainService - main.api.API - deps.helper.format_data - deps.validator.validate_input - vendor.logger.Logger - vendor.metrics.MetricsCollector - ``` - -### 📝 Current Limitations (Not Bugs) -1. **Cross-Namespace Dependencies**: Not resolved by AST parser - - Import statements are parsed - - But dependencies not linked across namespaces - - Future enhancement opportunity - -2. **Component-Level Namespacing**: Uses class/function names, not file names - - This is by design - - Provides more granular component identification - ---- - -## Component Breakdown - -### Main Namespace (7 components) -| Component | Type | File | -|-----------|------|------| -| `main.service.MainService` | Class | service.py | -| `main.api.API` | Class | api.py | -| `main.models.User` | Class | models.py | -| `main.models.Product` | Class | models.py | -| `main.controller.Controller` | Class | controller.py | -| `main.utils.sanitize` | Function | utils.py | -| `main.utils.format_error` | Function | utils.py | - -### Deps Namespace (5 components) -| Component | Type | File | -|-----------|------|------| -| `deps.cache.Cache` | Class | cache.py | -| `deps.helper.format_data` | Function | helper.py | -| `deps.helper.merge_dicts` | Function | helper.py | -| `deps.validator.validate_input` | Function | validator.py | -| `deps.validator.validate_email` | Function | validator.py | - -### Vendor Namespace (2 components) -| Component | Type | File | -|-----------|------|------| -| `vendor.logger.Logger` | Class | logger.py | -| `vendor.metrics.MetricsCollector` | Class | metrics.py | - ---- - -## Test Code Quality - -### Features -1. **Comprehensive Coverage**: Tests entire pipeline end-to-end -2. **Realistic Scenario**: Creates actual Python files with imports -3. **Detailed Assertions**: 9 distinct validation points -4. **Clear Output**: Step-by-step progress with visual indicators -5. **Self-Cleaning**: Automatic cleanup of test artifacts -6. **Error Handling**: Graceful failure with detailed error messages - -### Output Quality -- ✓ Clear section headers with separators -- ✓ Progress indicators (✓, ✗, ⚠) -- ✓ Detailed component listings -- ✓ Namespace breakdown -- ✓ Comprehensive summary - ---- - -## Running the Test - -```bash -cd /Users/michaelassraf/Documents/GitHub/CodeWiki -source .venv/bin/activate -python3 test-multi-path/integration_test.py -``` - -**Expected Exit Code**: `0` (success) - ---- - -## Conclusion - -The multi-path analysis feature is **fully functional** and **production-ready**: - -1. ✅ Configuration system supports multiple source paths -2. ✅ Path aggregation works correctly -3. ✅ All paths are analyzed -4. ✅ Components are correctly namespaced -5. ✅ No errors or warnings generated -6. ✅ Component counts match expectations - -**Recommendation**: Feature ready for production use with optional future enhancement of cross-namespace dependency resolution. - ---- - -## Next Steps - -### Optional Enhancements -1. **Cross-Namespace Dependency Resolution**: Extend AST parser to resolve imports across namespaces -2. **Visualization**: Add dependency graph visualization for multi-path projects -3. **CLI Flag**: Add `--multi-path` flag for explicit multi-path mode indication - -### Documentation Updates -- Update README with multi-path examples -- Add multi-path configuration guide -- Document namespace format conventions diff --git a/test-multi-path/README.md b/test-multi-path/README.md deleted file mode 100644 index a74a6ba5..00000000 --- a/test-multi-path/README.md +++ /dev/null @@ -1,267 +0,0 @@ -# Multi-Path Integration Tests - -End-to-end integration tests for CodeWiki's multi-path analysis feature. - -## Quick Start - -```bash -cd /Users/michaelassraf/Documents/GitHub/CodeWiki -source .venv/bin/activate -python3 test-multi-path/integration_test.py -``` - -**Expected Output**: `✅ INTEGRATION TEST PASSED` with all 9 assertions passing. - ---- - -## Test Files - -| File | Purpose | -|------|---------| -| `integration_test.py` | Complete end-to-end integration test (executable) | -| `test_multi_path.py` | Unit tests for Config (legacy) | -| `INTEGRATION_TEST_RESULTS.md` | Detailed test results and findings | -| `README.md` | This file | - ---- - -## What the Integration Test Does - -### 1. Environment Setup -- Creates temporary test directory -- Generates 10 Python files across 3 namespaces -- Creates realistic cross-path imports - -### 2. Configuration -- Creates `Config` with `additional_source_paths` -- Verifies multi-path mode detection - -### 3. Execution -- Runs `DependencyGraphBuilder` -- Parses all source paths -- Generates dependency graph - -### 4. Validation -- ✓ All paths analyzed -- ✓ Correct namespace prefixes -- ✓ Expected component counts -- ✓ No errors or warnings - -### 5. Output -- Detailed progress tracking -- Component listings -- Namespace breakdown -- Validation summary - ---- - -## Test Structure - -``` -test-multi-path/ -├── integration_test.py # Main integration test (598 lines) -├── test_multi_path.py # Legacy unit tests -├── INTEGRATION_TEST_RESULTS.md # Test results documentation -└── README.md # This file -``` - ---- - -## Test Stages - -The integration test runs **8 distinct stages**: - -1. **Setup Test Environment** - Create temp directories and Python files -2. **Creating Configuration** - Initialize Config with additional paths -3. **Validating Paths** - Verify all paths exist -4. **Executing Dependency Graph Builder** - Run analysis pipeline -5. **Verifying Component Namespaces** - Check namespace prefixes -6. **Verifying Cross-Path Dependencies** - Check import resolution -7. **Checking for Warnings** - Validate clean execution -8. **Verifying Component Counts** - Final count validation - ---- - -## Expected Test Output - -``` -================================================================================ -CODEWIKI MULTI-PATH INTEGRATION TEST -================================================================================ - -Testing complete end-to-end pipeline: - 1. Test environment setup - 2. Multi-path configuration - 3. Dependency parsing - 4. Namespace verification - 5. Cross-path dependency resolution - 6. Validation and reporting - -================================================================================ -STEP 1: Setting Up Test Environment -================================================================================ -📁 Test directory: /var/folders/.../codewiki_integration_xxx - ✓ Created main/ → ... - ✓ Created deps/ → ... - ✓ Created vendor/ → ... - ✓ Created 5 files in main/ - ✓ Created 3 files in deps/ - ✓ Created 2 files in vendor/ - -[... stages 2-8 ...] - -================================================================================ -VALIDATION SUMMARY -================================================================================ - -Total Assertions: 9 - ✓ Passed: 9 - -================================================================================ -✅ INTEGRATION TEST PASSED -================================================================================ -``` - ---- - -## Test Results - -### ✅ All Assertions Passing -1. Test directories created -2. Config created with 3 source paths -3. All paths validated successfully -4. Multi-path mode detected -5. All expected namespaces present -6. Component counts match expectations -7. Multi-path mode working (dependencies optional) -8. No warnings generated -9. Component counts match expectations - ---- - -## Key Findings - -### Namespace Format -Components are identified as: `..` - -**Examples:** -``` -main.service.MainService -main.api.API -deps.helper.format_data -deps.validator.validate_input -vendor.logger.Logger -vendor.metrics.MetricsCollector -``` - -### Component Counts -- **main**: 7 components (5 classes + 2 functions) -- **deps**: 5 components (1 class + 4 functions) -- **vendor**: 2 components (2 classes) -- **Total**: 14 components - -### Current Limitations -- Cross-namespace dependency resolution not implemented in AST parser -- Import statements parsed but not resolved across namespaces -- Future enhancement opportunity (not a bug) - ---- - -## Troubleshooting - -### Test Fails with Import Error -```bash -# Ensure virtual environment is activated -source .venv/bin/activate - -# Verify dependencies installed -pip install -r requirements.txt -``` - -### Test Fails with Permission Error -```bash -# Make test executable -chmod +x test-multi-path/integration_test.py -``` - -### Test Hangs -```bash -# Check for stuck processes -ps aux | grep python - -# Kill if necessary -pkill -f "python.*integration_test" -``` - ---- - -## Running Individual Tests - -### Run Integration Test -```bash -python3 test-multi-path/integration_test.py -``` - -### Run Legacy Unit Tests -```bash -python3 test-multi-path/test_multi_path.py -``` - ---- - -## Exit Codes - -| Code | Meaning | -|------|---------| -| `0` | All tests passed | -| `1` | Some assertions failed | -| `2` | Test crashed with exception | - ---- - -## Test Coverage - -The integration test covers: -- ✓ Configuration initialization -- ✓ Path validation -- ✓ Multi-path detection -- ✓ Dependency graph building -- ✓ Component parsing -- ✓ Namespace assignment -- ✓ Component counting -- ✓ Error handling -- ✓ Cleanup - -**Coverage**: 100% of multi-path pipeline - ---- - -## Development Notes - -### Adding New Assertions -1. Add assertion in appropriate stage method -2. Call `self._assert("Description", condition)` -3. Update expected counts in validation summary - -### Adding New Test Files -1. Create files in `_create_*_files()` methods -2. Update expected component counts -3. Add expected namespace entries - -### Debugging Failed Tests -1. Check `/tmp/integration_test.log` for detailed output -2. Review component IDs in "DETAILED OUTPUT SUMMARY" -3. Verify namespace breakdown matches expectations - ---- - -## References - -- **Feature Documentation**: See `INTEGRATION_TEST_RESULTS.md` -- **Implementation**: `/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/config.py` -- **Parser**: `/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/be/dependency_analyzer/` - ---- - -*Last Updated: February 2, 2026* -*Test Status: ✅ PASSING (9/9 assertions)* diff --git a/test-multi-path/TEST_RESULTS.md b/test-multi-path/TEST_RESULTS.md deleted file mode 100644 index d79eb0fe..00000000 --- a/test-multi-path/TEST_RESULTS.md +++ /dev/null @@ -1,326 +0,0 @@ -# Multi-Path Feature Test Results - -## Summary - -**Status:** ✅ **ALL TESTS PASSED** (7/7) - -The CodeWiki multi-path feature is working correctly and ready for production use. - -## Test Execution - -```bash -cd /Users/michaelassraf/Documents/GitHub/CodeWiki/test-multi-path -bash run_tests.sh -``` - -## Results Breakdown - -### ✅ Test 1: Single Path Analysis (Backward Compatibility) -**Status:** PASSED - -**What it tests:** -- Verifies that single-path analysis still works correctly -- Ensures backward compatibility with existing workflows - -**Results:** -- Found 2 components from `main/` directory -- Correctly identified `controller.py` (controller.APIController) -- Correctly identified `service.py` (service.MainService) - -**Conclusion:** Single-path mode works identically to original behavior. - ---- - -### ✅ Test 2: Multiple Paths With Unique Components -**Status:** PASSED - -**What it tests:** -- Validates multi-path analysis functionality -- Ensures all source directories are analyzed - -**Results:** -- Successfully analyzed 3 directories: `main/`, `deps/`, `external/` -- Found 9 total components across all paths: - - `main/`: 2 components (service.py, controller.py) - - `deps/`: 5 components (3 in helper.py, 2 in utils.py) - - `external/`: 2 components (plugin.py) - -**Conclusion:** Multi-path analysis correctly discovers components from all source directories. - ---- - -### ✅ Test 3: Component ID Namespacing -**Status:** PASSED - -**What it tests:** -- Ensures no component ID collisions between paths -- Verifies proper namespacing of component IDs - -**Results:** -- All 7 component IDs are unique -- Component IDs include path prefixes: - - `main.controller.APIController` - - `main.service.MainService` - - `deps.helper.validate_input` - - `deps.helper.process_data` - - `deps.helper.format_output` - - `deps.utils.sanitize_string` - - `deps.utils.calculate_hash` - -**Conclusion:** Namespacing prevents ID collisions and provides clear source attribution. - ---- - -### ✅ Test 4: Cross-Path Dependencies -**Status:** PASSED (with informational warnings) - -**What it tests:** -- Verifies dependency tracking across path boundaries -- Checks that imports between different source directories are detected - -**Results:** -- Found 7 components and 2 dependency edges -- Note: Warning about missing cross-path dependencies is informational - - The test data has imports but they may not create strong dependencies - - The dependency detection is working correctly - -**Conclusion:** Dependency tracking works across multiple source paths. - ---- - -### ✅ Test 5: Invalid Path Handling -**Status:** PASSED - -**What it tests:** -- Tests error handling for non-existent paths -- Verifies validation logic - -**Results:** -- Correctly raised `ValueError` for invalid path: - ``` - "Additional source path #1 does not exist: /nonexistent/path/that/does/not/exist" - ``` - -**Conclusion:** Validation correctly rejects invalid paths with clear error messages. - ---- - -### ✅ Test 6: Empty Additional Paths -**Status:** PASSED - -**What it tests:** -- Tests behavior with empty additional paths list -- Ensures no errors with edge case input - -**Results:** -- Successfully analyzed main path only -- Found 2 components from `main/` directory -- Behaved identically to single-path mode - -**Conclusion:** Empty additional paths list works correctly (graceful degradation to single-path). - ---- - -### ✅ Test 7: Relative vs Absolute Paths -**Status:** PASSED - -**What it tests:** -- Validates path resolution logic -- Ensures absolute paths work correctly - -**Results:** -- Successfully analyzed with absolute paths -- Found 7 components across multiple directories -- Path normalization handled correctly - -**Conclusion:** Absolute paths work correctly. Relative path support would require additional resolution logic. - ---- - -## Feature Validation - -### Core Capabilities ✅ - -1. **Multi-Path Analysis** - - ✅ Analyzes multiple source directories in single run - - ✅ Discovers all components from all paths - - ✅ Works with 2+ additional source paths - -2. **Component Namespacing** - - ✅ Prevents ID collisions with path prefixes - - ✅ Clear attribution of components to source paths - - ✅ Namespace format: `{source_dir}.{module}.{component}` - -3. **Backward Compatibility** - - ✅ Single-path mode unchanged - - ✅ Empty additional paths gracefully handled - - ✅ Existing workflows unaffected - -4. **Error Handling** - - ✅ Invalid paths detected and rejected - - ✅ Clear error messages for debugging - - ✅ Validation runs before analysis - -### Test Coverage - -- ✅ **Functionality:** All core features tested -- ✅ **Edge Cases:** Invalid inputs, empty lists, single path -- ✅ **Integration:** Full end-to-end analysis pipeline -- ✅ **Compatibility:** Backward compatibility verified - ---- - -## Test Data - -### Structure -``` -test-multi-path/ -├── main/ # Main codebase (2 files) -│ ├── service.py # Service class with imports from deps -│ └── controller.py # Controller using service -├── deps/ # Dependency package (2 files) -│ ├── helper.py # Helper utilities (3 functions) -│ └── utils.py # Additional utilities (2 functions) -├── external/ # External plugin (1 file) -│ └── plugin.py # Plugin interface (2 classes) -├── test_multi_path.py # Test runner -├── debug_parser.py # Debug script -└── run_tests.sh # Test execution script -``` - -### Cross-Path Dependencies -- `main/service.py` imports from `deps/helper.py` - ```python - from deps.helper import process_data, validate_input - ``` -- Creates realistic multi-path scenario - ---- - -## Configuration Example - -```python -from codewiki.src.config import Config - -config = Config( - repo_path="/path/to/main/codebase", - additional_source_paths=[ - "/path/to/dependencies", - "/path/to/plugins", - "/path/to/shared/library" - ], - # ... other required config fields -) - -# Validate paths before use -config.validate_source_paths() - -# Check if multi-path mode is enabled -if config.is_multi_path_mode(): - print(f"Analyzing {len(config.all_source_paths)} source paths") -``` - ---- - -## Performance Characteristics - -Based on test execution: - -- **Single Path Analysis:** < 1 second for 2 components -- **Multi-Path Analysis:** < 2 seconds for 9 components across 3 paths -- **Memory Usage:** Minimal overhead for multiple paths -- **Scalability:** Linear growth with number of paths - ---- - -## Known Limitations - -1. **Relative Paths:** Currently only absolute paths are fully supported - - Workaround: Convert relative to absolute before passing to Config - - Enhancement: Add automatic relative path resolution - -2. **Pattern Application:** File patterns (include/exclude) not yet tested across paths - - Test 5 placeholder exists for future implementation - - Current patterns apply to all paths equally - -3. **Circular Dependencies:** Not specifically tested - - Existing dependency graph logic should handle them - - No multi-path specific issues expected - ---- - -## Recommendations - -### For Production Use - -1. ✅ **Ready to use** - All core functionality working -2. ✅ **Stable** - Backward compatible, no breaking changes -3. ✅ **Validated** - Comprehensive test coverage - -### For Future Enhancements - -1. **Relative Path Resolution** - - Add automatic resolution relative to main repo_path - - Or relative to current working directory - -2. **Pattern Filtering Tests** - - Verify include/exclude patterns work across all paths - - Test pattern precedence and inheritance - -3. **Performance Tests** - - Benchmark with larger codebases (100+ files per path) - - Test with 10+ source paths - - Profile memory usage with large dependency graphs - -4. **Documentation** - - Add multi-path examples to user docs - - Document namespacing behavior - - Provide migration guide from single-path - ---- - -## Conclusion - -The multi-path feature is **production-ready** with: - -- ✅ All 7 tests passing -- ✅ Core functionality working correctly -- ✅ Backward compatibility maintained -- ✅ Proper error handling -- ✅ Clear component namespacing -- ✅ Cross-path dependency tracking - -The feature successfully enables CodeWiki to analyze codebases distributed across multiple directories, which is essential for: - -- Monorepos with shared libraries -- Projects with external dependencies -- Multi-package Python projects -- Documentation of interconnected codebases - -**Recommendation:** Proceed with production deployment. - ---- - -## Files Modified - -1. **`/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/config.py`** - - Added: `all_source_paths` property - - Added: `validate_source_paths()` method - - Added: `is_multi_path_mode()` method - - Added: `additional_source_paths` field - -2. **`/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/be/dependency_analyzer/dependency_graphs_builder.py`** - - Updated: Use `config.all_source_paths` property - - Updated: Log multi-path mode when enabled - -3. **`/Users/michaelassraf/Documents/GitHub/CodeWiki/codewiki/src/be/dependency_analyzer/ast_parser.py`** - - Already supported multi-path in constructor - - Already implemented `_parse_multiple_repositories()` - - Already implemented component namespacing - ---- - -**Test Date:** February 2, 2026 -**Test Environment:** Python 3.14, CodeWiki venv -**Test Duration:** ~2 seconds total -**Test Status:** ✅ ALL PASSED diff --git a/test-multi-path/debug_parser.py b/test-multi-path/debug_parser.py deleted file mode 100644 index 60a62f1d..00000000 --- a/test-multi-path/debug_parser.py +++ /dev/null @@ -1,49 +0,0 @@ -#!/usr/bin/env python3 -"""Debug script to check if parser finds files.""" - -import sys -import os -from pathlib import Path - -# Add CodeWiki to path -CODEWIKI_ROOT = Path(__file__).parent.parent -sys.path.insert(0, str(CODEWIKI_ROOT)) - -from codewiki.src.be.dependency_analyzer.ast_parser import DependencyParser - -# Test paths -main_path = Path(__file__).parent / "main" -deps_path = Path(__file__).parent / "deps" - -print(f"Main path: {main_path}") -print(f"Deps path: {deps_path}") -print(f"Main exists: {main_path.exists()}") -print(f"Deps exists: {deps_path.exists()}") - -print("\nFiles in main:") -for f in main_path.glob("*.py"): - print(f" - {f.name}") - -print("\nFiles in deps:") -for f in deps_path.glob("*.py"): - print(f" - {f.name}") - -print("\n" + "="*70) -print("Testing single path parser...") -print("="*70) - -parser_single = DependencyParser(str(main_path)) -components_single = parser_single.parse_repository() -print(f"\nSingle path found {len(components_single)} components") -for comp_id, comp in list(components_single.items())[:5]: - print(f" - {comp_id}: {comp.component_type}") - -print("\n" + "="*70) -print("Testing multi-path parser...") -print("="*70) - -parser_multi = DependencyParser([str(main_path), str(deps_path)]) -components_multi = parser_multi.parse_repository() -print(f"\nMulti-path found {len(components_multi)} components") -for comp_id, comp in list(components_multi.items())[:10]: - print(f" - {comp_id}: {comp.component_type}") diff --git a/test-multi-path/deps/.helper.md b/test-multi-path/deps/.helper.md deleted file mode 100644 index f1adb3d4..00000000 --- a/test-multi-path/deps/.helper.md +++ /dev/null @@ -1,38 +0,0 @@ - -Helper utilities for validating, processing, and formatting dictionary-based data structures. Intended as a shared dependency module used alongside the main codebase. - -## Key Components - -| Function | Description | -|---|---| -| `validate_input(data)` | Checks that input is a `dict` containing required keys `"type"` and `"value"` | -| `process_data(data)` | Validates and transforms input, returning a new dict with uppercased type, stripped value, and a fixed timestamp | -| `format_output(data)` | Renders a processed data dict as a human-readable string | - -## Usage Example - -```python -from helper import validate_input, process_data, format_output - -raw = {"type": "alert", "value": " disk usage high "} - -# Validate before processing -if validate_input(raw): - result = process_data(raw) - print(result) - # { - # "original": {"type": "alert", "value": " disk usage high "}, - # "processed_type": "ALERT", - # "processed_value": "disk usage high", - # "timestamp": "2024-01-01T00:00:00Z" - # } - - print(format_output(result)) - # [ALERT] disk usage high -``` - -## Notes - -- `process_data` raises `ValueError` if validation fails — always validate first or handle the exception. -- `format_output` uses `.get()` with safe defaults (`"UNKNOWN"` / empty string), so it won't raise on partial data. -- The `timestamp` field is currently hardcoded; replace with `datetime.utcnow().isoformat()` for production use. \ No newline at end of file diff --git a/test-multi-path/deps/.utils.md b/test-multi-path/deps/.utils.md deleted file mode 100644 index ee97200c..00000000 --- a/test-multi-path/deps/.utils.md +++ /dev/null @@ -1,27 +0,0 @@ - -Additional utility functions for string sanitization and simple hash calculation. - -## Key Components - -- **`sanitize_string(text: str) -> str`** — Strips non-alphanumeric characters from a string, preserving spaces. -- **`calculate_hash(data: str) -> int`** — Returns a simple 8-digit bounded hash of a string using Python's built-in `hash()`. - -## Usage Example - -```python -from utils import sanitize_string, calculate_hash - -# Sanitize a string -clean = sanitize_string("Hello, World! #2024") -# clean → "Hello World 2024" - -# Calculate a hash -hash_val = calculate_hash("my-data-key") -# hash_val → integer in range [0, 99_999_999] -``` - -> **Note:** `calculate_hash` uses Python's built-in `hash()`, which is non-deterministic across processes by default (due to hash randomization). Do not use this for persistent or cross-process key lookups. - -## Source - -[`utils.py`](https://github.com/flamingo-stack/CodeWiki/blob/main/utils.py) \ No newline at end of file diff --git a/test-multi-path/deps/helper.py b/test-multi-path/deps/helper.py deleted file mode 100644 index 6bcb3884..00000000 --- a/test-multi-path/deps/helper.py +++ /dev/null @@ -1,59 +0,0 @@ -""" -Helper utilities for data processing. - -This module is in a separate dependency package that should be analyzed -alongside the main codebase. -""" - -from typing import Any, Dict - - -def validate_input(data: Dict[str, Any]) -> bool: - """Validate input data structure. - - Args: - data: Input data to validate - - Returns: - True if valid, False otherwise - """ - if not isinstance(data, dict): - return False - - required_keys = ["type", "value"] - return all(key in data for key in required_keys) - - -def process_data(data: Dict[str, Any]) -> Dict[str, Any]: - """Process input data and return transformed result. - - Args: - data: Input data to process - - Returns: - Processed data - """ - if not validate_input(data): - raise ValueError("Invalid input data") - - # Transform data - result = { - "original": data, - "processed_type": data["type"].upper(), - "processed_value": str(data["value"]).strip(), - "timestamp": "2024-01-01T00:00:00Z" - } - - return result - - -def format_output(data: Dict[str, Any]) -> str: - """Format processed data for output. - - Args: - data: Data to format - - Returns: - Formatted string representation - """ - return f"[{data.get('processed_type', 'UNKNOWN')}] {data.get('processed_value', '')}" diff --git a/test-multi-path/deps/utils.py b/test-multi-path/deps/utils.py deleted file mode 100644 index 0ce15091..00000000 --- a/test-multi-path/deps/utils.py +++ /dev/null @@ -1,27 +0,0 @@ -""" -Additional utility functions. -""" - - -def sanitize_string(text: str) -> str: - """Remove special characters from string. - - Args: - text: Input text - - Returns: - Sanitized text - """ - return "".join(c for c in text if c.isalnum() or c.isspace()) - - -def calculate_hash(data: str) -> int: - """Calculate simple hash of string data. - - Args: - data: Input data - - Returns: - Hash value - """ - return hash(data) % (10 ** 8) diff --git a/test-multi-path/external/.plugin.md b/test-multi-path/external/.plugin.md deleted file mode 100644 index c552583e..00000000 --- a/test-multi-path/external/.plugin.md +++ /dev/null @@ -1,42 +0,0 @@ - -Defines the plugin interface and a concrete data-processing implementation for the CodeWiki external plugin system. - -## Key Components - -| Name | Type | Description | -|------|------|-------------| -| `PluginInterface` | Abstract Base Class | Base interface defining the plugin contract with `initialize()` and `execute()` methods | -| `DataPlugin` | Class | Concrete plugin implementation for data processing; inherits from `PluginInterface` | -| `initialize()` | Method | Sets up the plugin and returns `True` on success; must be called before `execute()` | -| `execute(context)` | Method | Runs plugin logic against a context dict and returns a result dict | - -## Usage Example - -```python -from plugin import DataPlugin - -# Instantiate and initialize the plugin -plugin = DataPlugin(name="my_data_plugin") -plugin.initialize() - -# Execute with a context payload -result = plugin.execute(context={"data": {"key": "value"}}) -print(result) -# Output: {'plugin': 'my_data_plugin', 'result': "Processed: {'key': 'value'}"} -``` - -> **Note:** Calling `execute()` before `initialize()` raises a `RuntimeError`. Always call `initialize()` first. - -## Extending the Interface - -```python -from plugin import PluginInterface - -class CustomPlugin(PluginInterface): - def initialize(self) -> bool: - # Custom setup logic - return True - - def execute(self, context: dict) -> dict: - return {"plugin": "custom", "result": context} -``` \ No newline at end of file diff --git a/test-multi-path/external/plugin.py b/test-multi-path/external/plugin.py deleted file mode 100644 index 6c961718..00000000 --- a/test-multi-path/external/plugin.py +++ /dev/null @@ -1,64 +0,0 @@ -""" -External plugin module. - -This is from a third package that should also be analyzed. -""" - - -class PluginInterface: - """Base interface for plugins.""" - - def initialize(self) -> bool: - """Initialize the plugin. - - Returns: - True if successful - """ - raise NotImplementedError("Subclasses must implement initialize()") - - def execute(self, context: dict) -> dict: - """Execute plugin logic. - - Args: - context: Execution context - - Returns: - Result data - """ - raise NotImplementedError("Subclasses must implement execute()") - - -class DataPlugin(PluginInterface): - """Plugin for data processing.""" - - def __init__(self, name: str): - """Initialize data plugin. - - Args: - name: Plugin name - """ - self.name = name - self.initialized = False - - def initialize(self) -> bool: - """Initialize the plugin.""" - self.initialized = True - return True - - def execute(self, context: dict) -> dict: - """Execute data processing. - - Args: - context: Execution context with 'data' key - - Returns: - Processed result - """ - if not self.initialized: - raise RuntimeError("Plugin not initialized") - - data = context.get("data", {}) - return { - "plugin": self.name, - "result": f"Processed: {data}" - } diff --git a/test-multi-path/integration_test.py b/test-multi-path/integration_test.py deleted file mode 100755 index aec0184e..00000000 --- a/test-multi-path/integration_test.py +++ /dev/null @@ -1,617 +0,0 @@ -#!/usr/bin/env python3 -""" -End-to-End Integration Test for Multi-Path Analysis Pipeline - -Tests the complete workflow: -1. Test environment setup with multiple source paths -2. Sample Python files with cross-path imports -3. Full pipeline execution (Config → DependencyParser) -4. Comprehensive assertions on all outputs -5. Detailed validation reporting -""" - -import os -import sys -import shutil -import tempfile -from pathlib import Path -from typing import Dict, List, Set - -# Add CodeWiki to path -CODEWIKI_ROOT = Path(__file__).parent.parent -sys.path.insert(0, str(CODEWIKI_ROOT)) - -from codewiki.src.config import Config -from codewiki.src.be.dependency_analyzer import DependencyGraphBuilder - - -class TestResults: - """Standardized accumulator for integration test results""" - - def __init__(self): - self.tests = [] - - def add_test(self, name: str, passed: bool, details: str = "") -> None: - """Record a single test result""" - self.tests.append({ - "name": name, - "passed": passed, - "details": details - }) - - def print_summary(self) -> None: - """Print final validation summary""" - print("\n" + "="*80) - print("VALIDATION SUMMARY") - print("="*80) - - total = len(self.tests) - passed_tests = [t for t in self.tests if t["passed"]] - failed_tests = [t for t in self.tests if not t["passed"]] - warning_tests = [t for t in self.tests if t["passed"] and t["details"]] - - passed = len(passed_tests) - failed = len(failed_tests) - - print(f"\nTotal Assertions: {total}") - print(f" ✓ Passed: {passed}") - if failed > 0: - print(f" ✗ Failed: {failed}") - - if failed > 0: - print("\nFailed Assertions:") - for t in failed_tests: - msg = f" ✗ {t['name']}" - if t["details"]: - msg += f" - {t['details']}" - print(msg) - - if warning_tests: - print("\nWarnings:") - for t in warning_tests: - print(f" ⚠ {t['details']}") - - print("\n" + "="*80) - if failed == 0: - print("✅ INTEGRATION TEST PASSED") - else: - print("❌ INTEGRATION TEST FAILED") - print("="*80) - - def all_passed(self) -> bool: - return all(t["passed"] for t in self.tests) - - -class IntegrationTestRunner: - """Manages end-to-end integration test execution""" - - def __init__(self): - self.test_dir = None - self.main_path = None - self.deps_path = None - self.vendor_path = None - self.config = None - self.builder = None - self.components = None - self.leaf_nodes = None - self.results = TestResults() - - def setup_test_environment(self) -> None: - """Create test directory structure with sample files""" - print("\n" + "="*80) - print("STEP 1: Setting Up Test Environment") - print("="*80) - - # Create temporary test directory - self.test_dir = Path(tempfile.mkdtemp(prefix="codewiki_integration_")) - print(f"📁 Test directory: {self.test_dir}") - - # Create three source paths - self.main_path = self.test_dir / "main" - self.deps_path = self.test_dir / "deps" - self.vendor_path = self.test_dir / "vendor" - - self.main_path.mkdir() - self.deps_path.mkdir() - self.vendor_path.mkdir() - - print(f" ✓ Created main/ → {self.main_path}") - print(f" ✓ Created deps/ → {self.deps_path}") - print(f" ✓ Created vendor/ → {self.vendor_path}") - - # Create sample files in main/ - self._create_main_files() - - # Create sample files in deps/ - self._create_deps_files() - - # Create sample files in vendor/ - self._create_vendor_files() - - self.results.add_test("Test directories created", True) - - def _create_main_files(self) -> None: - """Create sample Python files in main/ directory""" - - # main/service.py - imports from deps and vendor - (self.main_path / "service.py").write_text(""" -# Main service that uses dependencies from other paths -from deps.helper import format_data -from vendor.logger import Logger - -class MainService: - def __init__(self): - self.logger = Logger('MainService') - - def process(self, data): - formatted = format_data(data) - self.logger.info(f"Processed: {formatted}") - return formatted -""") - - # main/api.py - imports from main/ and deps/ - (self.main_path / "api.py").write_text(""" -# API layer that uses service and validation -from service import MainService -from deps.validator import validate_input - -class API: - def __init__(self): - self.service = MainService() - - def handle_request(self, data): - if validate_input(data): - return self.service.process(data) - return None -""") - - # main/models.py - standalone model definitions - (self.main_path / "models.py").write_text(""" -# Data models -class User: - def __init__(self, name, email): - self.name = name - self.email = email - -class Product: - def __init__(self, id, name, price): - self.id = id - self.name = name - self.price = price -""") - - # main/controller.py - uses api and models - (self.main_path / "controller.py").write_text(""" -# Controller layer -from api import API -from models import User, Product - -class Controller: - def __init__(self): - self.api = API() - - def create_user(self, name, email): - user = User(name, email) - return self.api.handle_request(user) -""") - - # main/utils.py - utility functions - (self.main_path / "utils.py").write_text(""" -# Utility functions -def sanitize(text): - return text.strip().lower() - -def format_error(message): - return f"ERROR: {message}" -""") - - print(f" ✓ Created 5 files in main/: service.py, api.py, models.py, controller.py, utils.py") - - def _create_deps_files(self) -> None: - """Create sample Python files in deps/ directory""" - - # deps/helper.py - utility functions used by main - (self.deps_path / "helper.py").write_text(""" -# Helper utilities -def format_data(data): - if isinstance(data, dict): - return {k: str(v) for k, v in data.items()} - return str(data) - -def merge_dicts(*dicts): - result = {} - for d in dicts: - result.update(d) - return result -""") - - # deps/validator.py - validation logic - (self.deps_path / "validator.py").write_text(""" -# Input validation -from vendor.logger import Logger - -logger = Logger('Validator') - -def validate_input(data): - if data is None: - logger.error("Data is None") - return False - return True - -def validate_email(email): - return '@' in email and '.' in email -""") - - # deps/cache.py - caching layer - (self.deps_path / "cache.py").write_text(""" -# Simple cache implementation -class Cache: - def __init__(self): - self._data = {} - - def get(self, key): - return self._data.get(key) - - def set(self, key, value): - self._data[key] = value -""") - - print(f" ✓ Created 3 files in deps/: helper.py, validator.py, cache.py") - - def _create_vendor_files(self) -> None: - """Create sample Python files in vendor/ directory""" - - # vendor/logger.py - logging utility - (self.vendor_path / "logger.py").write_text(""" -# Logging utility -import sys - -class Logger: - def __init__(self, name): - self.name = name - - def info(self, message): - print(f"[INFO] {self.name}: {message}") - - def error(self, message): - print(f"[ERROR] {self.name}: {message}", file=sys.stderr) -""") - - # vendor/metrics.py - metrics collection - (self.vendor_path / "metrics.py").write_text(""" -# Metrics collection -from logger import Logger - -class MetricsCollector: - def __init__(self): - self.logger = Logger('Metrics') - self.counters = {} - - def increment(self, name): - self.counters[name] = self.counters.get(name, 0) + 1 - self.logger.info(f"Counter {name}: {self.counters[name]}") -""") - - print(f" ✓ Created 2 files in vendor/: logger.py, metrics.py") - - def create_config(self) -> None: - """Create Config with additional_source_paths""" - print("\n" + "="*80) - print("STEP 2: Creating Configuration") - print("="*80) - - output_dir = str(self.test_dir / "output") - - # Create config with main path as root, others as additional - self.config = Config( - repo_path=str(self.main_path), - output_dir=output_dir, - dependency_graph_dir=output_dir, - docs_dir=output_dir, - max_depth=5, - main_model="gpt-4", - cluster_model="gpt-4", - fallback_model="claude-3", - cluster_api_key="test", - main_api_key="test", - fallback_api_key="test", - additional_source_paths=[ - str(self.deps_path), - str(self.vendor_path) - ] - ) - - print(f"✓ Config created with root: {self.config.repo_path}") - print(f"✓ Additional paths: {len(self.config.additional_source_paths)}") - for i, path in enumerate(self.config.additional_source_paths, 1): - print(f" {i}. {path}") - - self.results.add_test("Config created with 3 source paths", True) - - def validate_paths(self) -> None: - """Validate all configured paths exist""" - print("\n" + "="*80) - print("STEP 3: Validating Paths") - print("="*80) - - # Validate root path - root_exists = Path(self.config.repo_path).exists() - print(f"{'✓' if root_exists else '✗'} Root path exists: {self.config.repo_path}") - - # Validate additional paths - all_valid = root_exists - for path in self.config.additional_source_paths: - exists = Path(path).exists() - print(f"{'✓' if exists else '✗'} Additional path exists: {path}") - all_valid = all_valid and exists - - self.results.add_test("All paths validated successfully", all_valid) - - def execute_dependency_parser(self) -> None: - """Run DependencyGraphBuilder with all configured paths""" - print("\n" + "="*80) - print("STEP 4: Executing Dependency Graph Builder") - print("="*80) - - self.builder = DependencyGraphBuilder(self.config) - - # Check if multi-path mode was detected - is_multi = self.config.is_multi_path_mode() - print(f"{'✓' if is_multi else '✗'} Multi-path mode detected: {is_multi}") - self.results.add_test("Multi-path mode detected", is_multi) - - # Build dependency graph - print("\nBuilding dependency graph...") - self.components, self.leaf_nodes = self.builder.build_dependency_graph() - - print(f"✓ Dependency graph building complete") - print(f" Total components found: {len(self.components)}") - print(f" Total leaf nodes found: {len(self.leaf_nodes)}") - - def verify_namespaces(self) -> None: - """Verify components have correct namespace prefixes""" - print("\n" + "="*80) - print("STEP 5: Verifying Component Namespaces") - print("="*80) - - components = self.components - - # Group components by top-level namespace (first part before .) - namespaces: Dict[str, List[str]] = {} - for comp_id in components.keys(): - # Extract top-level namespace (e.g., "main" from "main.service.MainService") - top_level = comp_id.split(".")[0] - namespaces.setdefault(top_level, []).append(comp_id) - - print(f"\nFound {len(namespaces)} top-level namespaces:") - for namespace, comp_ids in sorted(namespaces.items()): - print(f"\n Namespace '{namespace}': {len(comp_ids)} components") - for comp_id in sorted(comp_ids): - print(f" - {comp_id}") - - # Expected top-level namespaces - expected = {"main", "deps", "vendor"} - found = set(namespaces.keys()) - - missing = expected - found - unexpected = found - expected - - if missing: - print(f"\n✗ Missing namespaces: {missing}") - self.results.add_test("All expected namespaces present", False) - else: - print(f"\n✓ All expected namespaces present: {expected}") - self.results.add_test("All expected namespaces present", True) - - if unexpected: - print(f"⚠ Unexpected namespaces: {unexpected}") - self.results.add_test("No unexpected namespaces", True, f"Unexpected namespaces: {unexpected}") - - # Verify component counts - self._verify_namespace_counts(namespaces) - - def _verify_namespace_counts(self, namespaces: Dict[str, List[str]]) -> None: - """Verify expected component counts per namespace""" - # Expected counts at CLASS/FUNCTION level, not file level - # main: MainService, API, Product, User, Controller, format_error, sanitize = 7 - # deps: Cache, format_data, merge_dicts, validate_input, validate_email = 5 - # vendor: Logger, MetricsCollector = 2 - expected_counts = { - "main": 7, # Classes + functions - "deps": 5, # Classes + functions - "vendor": 2 # Classes only - } - - print("\nVerifying component counts (classes + functions):") - all_match = True - for namespace, expected in expected_counts.items(): - actual = len(namespaces.get(namespace, [])) - match = actual == expected - print(f" {'✓' if match else '✗'} Namespace '{namespace}': expected {expected}, found {actual}") - if not match: - print(f" Found components: {sorted(namespaces.get(namespace, []))}") - all_match = all_match and match - - self.results.add_test("Component counts match expectations", all_match) - - def verify_cross_path_dependencies(self) -> None: - """Verify cross-path dependencies are correctly resolved""" - print("\n" + "="*80) - print("STEP 6: Verifying Cross-Path Dependencies") - print("="*80) - - components = self.components - - # Find cross-namespace dependencies (using dot notation) - cross_deps: List[tuple] = [] - for comp_id, component in components.items(): - # Extract top-level namespace - comp_namespace = comp_id.split(".")[0] - - # Check if component has dependencies attribute - if hasattr(component, 'dependencies'): - for dep_id in component.dependencies: - # Extract top-level namespace from dependency - dep_namespace = dep_id.split(".")[0] - if dep_namespace != comp_namespace and dep_namespace in ["main", "deps", "vendor"]: - cross_deps.append((comp_id, dep_id)) - - print(f"\nFound {len(cross_deps)} cross-namespace dependencies:") - for source, target in sorted(cross_deps): - print(f" {source}") - print(f" → {target}") - - # NOTE: Cross-path dependency resolution is not currently implemented in AST parser - # This is expected behavior - dependencies are detected at import statement level - # but not resolved to specific components across namespaces - if len(cross_deps) > 0: - print(f"\n✓ Cross-path dependencies detected: {len(cross_deps)} found") - self.results.add_test("Cross-path dependencies detected", True) - else: - print(f"\n✓ No cross-path dependencies detected (expected - not implemented in AST parser yet)") - print(f" Note: Import statements are parsed but not resolved across namespaces") - self.results.add_test("Multi-path mode working (dependencies optional)", True) - - def verify_no_warnings(self) -> None: - """Verify no 'not found' warnings were generated""" - print("\n" + "="*80) - print("STEP 7: Checking for Warnings") - print("="*80) - - # Check if parser has warnings attribute - has_warnings = hasattr(self.builder, 'warnings') - - if not has_warnings: - print("✓ No warning tracking mechanism (expected behavior)") - self.results.add_test("No warnings generated", True) - return - - warnings = getattr(self.builder, 'warnings', []) - - if warnings: - print(f"⚠ Found {len(warnings)} warnings:") - for warning in warnings: - print(f" - {warning}") - self.results.add_test("No warnings generated", False) - else: - print("✓ No warnings generated") - self.results.add_test("No warnings generated", True) - - def verify_file_counts(self) -> None: - """Verify expected number of components were analyzed""" - print("\n" + "="*80) - print("STEP 8: Verifying Component Counts") - print("="*80) - - components = self.components - - # Count components per namespace - comp_counts: Dict[str, int] = {} - for comp_id in components.keys(): - namespace = comp_id.split(".")[0] - comp_counts[namespace] = comp_counts.get(namespace, 0) + 1 - - print("\nComponent counts by namespace:") - total = 0 - for namespace, count in sorted(comp_counts.items()): - print(f" {namespace}: {count} components") - total += count - - expected_total = 14 # 7 main + 5 deps + 2 vendor - print(f"\nTotal components analyzed: {total}") - print(f"Expected: {expected_total}") - - match = total == expected_total - self.results.add_test("Component counts match expectations", match) - - def print_detailed_output(self) -> None: - """Print comprehensive analysis output""" - print("\n" + "="*80) - print("DETAILED OUTPUT SUMMARY") - print("="*80) - - print("\n1. ALL COMPONENT IDs FOUND:") - print("-" * 40) - for comp_id in sorted(self.components.keys()): - print(f" {comp_id}") - - print("\n2. CROSS-NAMESPACE DEPENDENCIES:") - print("-" * 40) - cross_deps = [] - for comp_id, component in self.components.items(): - comp_namespace = comp_id.split(".")[0] - if hasattr(component, 'dependencies'): - for dep_id in component.dependencies: - dep_namespace = dep_id.split(".")[0] - if dep_namespace != comp_namespace and dep_namespace in ["main", "deps", "vendor"]: - cross_deps.append((comp_id, dep_id)) - - if cross_deps: - for source, target in sorted(cross_deps): - print(f" {source} → {target}") - else: - print(" (none detected)") - - print("\n3. NAMESPACE → COMPONENT COUNT:") - print("-" * 40) - namespaces: Dict[str, int] = {} - for comp_id in self.components.keys(): - namespace = comp_id.split(".")[0] - namespaces[namespace] = namespaces.get(namespace, 0) + 1 - - for namespace, count in sorted(namespaces.items()): - print(f" {namespace}: {count} components") - - def cleanup(self) -> None: - """Clean up test directory""" - if self.test_dir and self.test_dir.exists(): - shutil.rmtree(self.test_dir) - print(f"\n🧹 Cleaned up test directory: {self.test_dir}") - - def run(self) -> int: - """Execute complete integration test""" - try: - self.setup_test_environment() - self.create_config() - self.validate_paths() - self.execute_dependency_parser() - self.verify_namespaces() - self.verify_cross_path_dependencies() - self.verify_no_warnings() - self.verify_file_counts() - self.print_detailed_output() - self.results.print_summary() - - return 0 if self.results.all_passed() else 1 - - except Exception as e: - print(f"\n❌ INTEGRATION TEST CRASHED: {e}") - import traceback - traceback.print_exc() - return 2 - - finally: - self.cleanup() - - -def main(): - """Main entry point""" - print("\n" + "="*80) - print("CODEWIKI MULTI-PATH INTEGRATION TEST") - print("="*80) - print("\nTesting complete end-to-end pipeline:") - print(" 1. Test environment setup") - print(" 2. Multi-path configuration") - print(" 3. Dependency parsing") - print(" 4. Namespace verification") - print(" 5. Cross-path dependency resolution") - print(" 6. Validation and reporting") - - runner = IntegrationTestRunner() - exit_code = runner.run() - - sys.exit(exit_code) - - -if __name__ == "__main__": - main() diff --git a/test-multi-path/main/.controller.md b/test-multi-path/main/.controller.md deleted file mode 100644 index d788ceaf..00000000 --- a/test-multi-path/main/.controller.md +++ /dev/null @@ -1,37 +0,0 @@ - -Handles HTTP-style API request routing by delegating business logic to `MainService` and tracking request counts. - -## Key Components - -- **`APIController`** — Main controller class that initializes a `MainService` instance and routes incoming requests to the appropriate handler. - - `__init__()` — Sets up the `MainService` dependency and initializes a request counter. - - `handle_request(endpoint, data)` — Routes requests based on endpoint path, returning processed results or health status. - -## Supported Endpoints - -| Endpoint | Behavior | -|---|---| -| `/process` | Delegates to `MainService.process_request()` | -| `/health` | Returns service health status and total request count | -| _(other)_ | Returns `{"error": "Unknown endpoint"}` | - -## Usage Example - -```python -from controller import APIController - -controller = APIController() - -# Process a request -response = controller.handle_request("/process", {"key": "value"}) - -# Check service health -health = controller.handle_request("/health", {}) -# {"status": "healthy", "requests": 1} -``` - -## Notes - -- Request count increments on **every** call to `handle_request`, including health checks. -- Unknown endpoints return an error dict rather than raising an exception — callers should check for the `"error"` key in the response. -- `MainService` is instantiated once at controller creation with the name `"api-service"`; it is not configurable via the constructor. \ No newline at end of file diff --git a/test-multi-path/main/.service.md b/test-multi-path/main/.service.md deleted file mode 100644 index abd10285..00000000 --- a/test-multi-path/main/.service.md +++ /dev/null @@ -1,51 +0,0 @@ - -Primary service component that manages application lifecycle and request processing, leveraging helper utilities from the `deps` package for validation and data transformation. - -## Key Components - -### `MainService` -The core service class responsible for managing service state and processing requests. - -| Method | Signature | Description | -|--------|-----------|-------------| -| `__init__` | `(name: str)` | Initializes the service with a name; sets `active = False` | -| `start` | `(config: dict) -> bool` | Validates config and activates the service | -| `process_request` | `(data: dict) -> dict` | Processes incoming data; raises `RuntimeError` if service is inactive | -| `stop` | `() -> None` | Deactivates the service | - -### External Dependencies -- `deps.helper.validate_input` — Validates configuration before startup -- `deps.helper.process_data` — Transforms raw request data - -## Usage Example - -```python -from service import MainService - -# Initialize and start the service -svc = MainService(name="my-service") - -config = {"timeout": 30, "retries": 3} -started = svc.start(config) - -if started: - # Process a request - response = svc.process_request({"key": "value"}) - print(response) - # { - # "status": "success", - # "service": "my-service", - # "result": - # } - - # Gracefully stop - svc.stop() -else: - print("Failed to start: invalid config") -``` - -## Notes - -- `process_request` raises `RuntimeError` if called before `start()` or after `stop()` -- `start()` returns `False` (without activating) when `validate_input` rejects the config -- Source: [`service.py`](https://github.com/flamingo-stack/CodeWiki/blob/main/service.py) \ No newline at end of file diff --git a/test-multi-path/main/controller.py b/test-multi-path/main/controller.py deleted file mode 100644 index 4ed6e950..00000000 --- a/test-multi-path/main/controller.py +++ /dev/null @@ -1,33 +0,0 @@ -""" -Controller component for handling API requests. -""" - -from service import MainService - - -class APIController: - """API Controller for routing requests.""" - - def __init__(self): - """Initialize the controller.""" - self.service = MainService("api-service") - self.request_count = 0 - - def handle_request(self, endpoint: str, data: dict) -> dict: - """Handle API request. - - Args: - endpoint: API endpoint path - data: Request data - - Returns: - Response data - """ - self.request_count += 1 - - if endpoint == "/process": - return self.service.process_request(data) - elif endpoint == "/health": - return {"status": "healthy", "requests": self.request_count} - else: - return {"error": "Unknown endpoint"} diff --git a/test-multi-path/main/service.py b/test-multi-path/main/service.py deleted file mode 100644 index 37343c02..00000000 --- a/test-multi-path/main/service.py +++ /dev/null @@ -1,60 +0,0 @@ -""" -Main service component. - -This is the primary service that uses helper utilities from the deps package. -""" - -from deps.helper import process_data, validate_input - - -class MainService: - """Main service class for the application.""" - - def __init__(self, name: str): - """Initialize the service. - - Args: - name: Service name - """ - self.name = name - self.active = False - - def start(self, config: dict) -> bool: - """Start the service with configuration. - - Args: - config: Configuration dictionary - - Returns: - True if started successfully - """ - if not validate_input(config): - return False - - self.active = True - return True - - def process_request(self, data: dict) -> dict: - """Process incoming request data. - - Args: - data: Request data to process - - Returns: - Processed response data - """ - if not self.active: - raise RuntimeError("Service not started") - - # Use helper from deps package - processed = process_data(data) - - return { - "status": "success", - "service": self.name, - "result": processed - } - - def stop(self): - """Stop the service.""" - self.active = False diff --git a/test-multi-path/run_tests.sh b/test-multi-path/run_tests.sh deleted file mode 100755 index b401f407..00000000 --- a/test-multi-path/run_tests.sh +++ /dev/null @@ -1,48 +0,0 @@ -#!/bin/bash -# Run multi-path tests with CodeWiki virtual environment - -set -e - -# Colors -GREEN='\033[0;32m' -RED='\033[0;31m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' # No Color - -echo -e "${BLUE}==================================================================${NC}" -echo -e "${BLUE}CodeWiki Multi-Path Feature Test Suite${NC}" -echo -e "${BLUE}==================================================================${NC}" - -# Find CodeWiki root -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -CODEWIKI_ROOT="$(dirname "$SCRIPT_DIR")" - -echo -e "\n${YELLOW}Setup:${NC}" -echo -e " Test directory: $SCRIPT_DIR" -echo -e " CodeWiki root: $CODEWIKI_ROOT" - -# Check virtual environment -if [ ! -d "$CODEWIKI_ROOT/venv" ]; then - echo -e "${RED}Error: Virtual environment not found at $CODEWIKI_ROOT/venv${NC}" - exit 1 -fi - -echo -e " Virtual env: $CODEWIKI_ROOT/venv" - -# Activate virtual environment and run tests -echo -e "\n${YELLOW}Running tests...${NC}\n" - -cd "$SCRIPT_DIR" -source "$CODEWIKI_ROOT/venv/bin/activate" -python test_multi_path.py - -exit_code=$? - -if [ $exit_code -eq 0 ]; then - echo -e "\n${GREEN}✓ Test suite completed successfully${NC}" -else - echo -e "\n${RED}✗ Test suite failed with exit code $exit_code${NC}" -fi - -exit $exit_code diff --git a/test-multi-path/test_multi_path.py b/test-multi-path/test_multi_path.py deleted file mode 100755 index fb8cb9fd..00000000 --- a/test-multi-path/test_multi_path.py +++ /dev/null @@ -1,546 +0,0 @@ -#!/usr/bin/env python3 -""" -Comprehensive test suite for CodeWiki multi-path feature. - -This script tests the ability to analyze code from multiple source directories -simultaneously while maintaining proper component namespacing and dependency tracking. - -Test Coverage: -1. Single path (backward compatibility) -2. Multiple paths with unique components -3. Component ID namespacing (no collisions) -4. Cross-path dependencies -5. Pattern application across paths -6. Invalid path handling -7. Empty additional paths -8. Relative vs absolute paths -""" - -import os -import sys -import json -import tempfile -import shutil -from pathlib import Path -from typing import Dict, List, Any - -# Add CodeWiki to path -CODEWIKI_ROOT = Path(__file__).parent.parent -sys.path.insert(0, str(CODEWIKI_ROOT)) - -from codewiki.src.config import Config -from codewiki.src.be.dependency_analyzer.dependency_graphs_builder import DependencyGraphBuilder - - -# ANSI color codes for output -class Colors: - GREEN = '\033[92m' - RED = '\033[91m' - YELLOW = '\033[93m' - BLUE = '\033[94m' - BOLD = '\033[1m' - END = '\033[0m' - - -class TestResults: - """Accumulates test results across the suite and prints a summary.""" - - def __init__(self): - self.results = {} - - def add_test(self, name: str, passed: bool): - """Record the result of a single test. - - Args: - name: Test name - passed: Whether the test passed - """ - self.results[name] = passed - - def print_summary(self) -> int: - """Print formatted summary of all recorded test results. - - Returns: - Exit code: 0 if all tests passed, 1 otherwise - """ - print_header("Test Results Summary") - - passed = sum(1 for r in self.results.values() if r) - total = len(self.results) - - for test_name, result in self.results.items(): - if result: - print_success(f"{test_name}") - else: - print_error(f"{test_name}") - - print(f"\n{Colors.BOLD}Total: {passed}/{total} tests passed{Colors.END}") - - if passed == total: - print(f"{Colors.GREEN}{Colors.BOLD}✓ All tests passed!{Colors.END}\n") - return 0 - else: - print(f"{Colors.RED}{Colors.BOLD}✗ Some tests failed{Colors.END}\n") - return 1 - - -def print_header(text: str): - """Print formatted test header.""" - print(f"\n{Colors.BOLD}{Colors.BLUE}{'='*70}{Colors.END}") - print(f"{Colors.BOLD}{Colors.BLUE}{text}{Colors.END}") - print(f"{Colors.BOLD}{Colors.BLUE}{'='*70}{Colors.END}\n") - - -def print_success(text: str): - """Print success message.""" - print(f"{Colors.GREEN}✓ {text}{Colors.END}") - - -def print_error(text: str): - """Print error message.""" - print(f"{Colors.RED}✗ {text}{Colors.END}") - - -def print_warning(text: str): - """Print warning message.""" - print(f"{Colors.YELLOW}⚠ {text}{Colors.END}") - - -def print_info(text: str): - """Print info message.""" - print(f" {text}") - - -def create_test_config( - repo_path: str, - output_dir: str, - additional_paths: List[str] = None -) -> Config: - """Create test configuration. - - Args: - repo_path: Main repository path - output_dir: Output directory - additional_paths: Additional source paths to analyze - - Returns: - Config instance - """ - return Config( - repo_path=repo_path, - output_dir=output_dir, - dependency_graph_dir=os.path.join(output_dir, "graphs"), - docs_dir=os.path.join(output_dir, "docs"), - max_depth=2, - main_model="gpt-4", - cluster_model="gpt-4", - fallback_model="gpt-3.5-turbo", - cluster_api_key="test-key", - main_api_key="test-key", - fallback_api_key="test-key", - cluster_base_url="http://localhost:4000", - main_base_url="http://localhost:4000", - fallback_base_url="http://localhost:4000", - additional_source_paths=additional_paths - ) - - -def test_single_path(): - """Test 1: Single path analysis (backward compatibility).""" - print_header("Test 1: Single Path Analysis (Backward Compatibility)") - - test_dir = Path(__file__).parent / "main" - output_dir = tempfile.mkdtemp(prefix="codewiki_test_single_") - - try: - config = create_test_config(str(test_dir), output_dir) - - print_info("Analyzing single path: main/") - builder = DependencyGraphBuilder(config) - components_dict, leaf_nodes = builder.build_dependency_graph() - - # components_dict is a Dict[str, Node], convert to dict for testing - components = {comp_id: comp.model_dump() for comp_id, comp in components_dict.items()} - if not components: - print_error("No components found") - return False - - print_success(f"Found {len(components)} components") - - # Check for expected files - expected_files = ["service.py", "controller.py"] - found_files = [] - for comp_id, comp_data in components.items(): - file_path = comp_data.get("file_path", "") - filename = os.path.basename(file_path) - if filename in expected_files: - found_files.append(filename) - print_info(f" - Found component: {filename} (ID: {comp_id})") - - if len(found_files) != len(expected_files): - print_error(f"Expected {len(expected_files)} files, found {len(found_files)}") - return False - - print_success("All expected components found in single path") - return True - - except Exception as e: - print_error(f"Test failed: {str(e)}") - import traceback - traceback.print_exc() - return False - finally: - shutil.rmtree(output_dir, ignore_errors=True) - - -def test_multiple_paths(): - """Test 2: Multiple paths with unique components.""" - print_header("Test 2: Multiple Paths With Unique Components") - - main_path = Path(__file__).parent / "main" - deps_path = Path(__file__).parent / "deps" - external_path = Path(__file__).parent / "external" - output_dir = tempfile.mkdtemp(prefix="codewiki_test_multi_") - - try: - config = create_test_config( - str(main_path), - output_dir, - additional_paths=[str(deps_path), str(external_path)] - ) - - print_info(f"Main path: {main_path}") - print_info(f"Additional paths: {deps_path}, {external_path}") - - builder = DependencyGraphBuilder(config) - components_dict, leaf_nodes = builder.build_dependency_graph() - - # components_dict is a Dict[str, Node], convert to dict for testing - components = {comp_id: comp.model_dump() for comp_id, comp in components_dict.items()} - if not components: - print_error("No components found") - return False - - print_success(f"Found {len(components)} total components") - - # Verify components from all paths - expected_components = { - "main": ["service.py", "controller.py"], - "deps": ["helper.py", "utils.py"], - "external": ["plugin.py"] - } - - found_by_path = {"main": [], "deps": [], "external": []} - - for comp_id, comp_data in components.items(): - file_path = comp_data.get("file_path", "") - filename = os.path.basename(file_path) - - if "main" in file_path: - found_by_path["main"].append(filename) - elif "deps" in file_path: - found_by_path["deps"].append(filename) - elif "external" in file_path: - found_by_path["external"].append(filename) - - all_found = True - for path_name, expected_files in expected_components.items(): - found_files = found_by_path[path_name] - print_info(f"\nPath: {path_name}/") - for filename in expected_files: - if filename in found_files: - print_success(f" ✓ {filename}") - else: - print_error(f" ✗ {filename} not found") - all_found = False - - if not all_found: - print_error("Some components missing from multi-path analysis") - return False - - print_success("\nAll components from all paths found") - return True - - except Exception as e: - print_error(f"Test failed: {str(e)}") - import traceback - traceback.print_exc() - return False - finally: - shutil.rmtree(output_dir, ignore_errors=True) - - -def test_component_namespacing(): - """Test 3: Component ID namespacing prevents collisions.""" - print_header("Test 3: Component ID Namespacing (No Collisions)") - - main_path = Path(__file__).parent / "main" - deps_path = Path(__file__).parent / "deps" - output_dir = tempfile.mkdtemp(prefix="codewiki_test_namespace_") - - try: - config = create_test_config( - str(main_path), - output_dir, - additional_paths=[str(deps_path)] - ) - - builder = DependencyGraphBuilder(config) - components_dict, leaf_nodes = builder.build_dependency_graph() - - # components_dict is a Dict[str, Node], convert to dict for testing - components = {comp_id: comp.model_dump() for comp_id, comp in components_dict.items()} - - # Check for ID uniqueness - component_ids = list(components.keys()) - if len(component_ids) != len(set(component_ids)): - print_error("Duplicate component IDs detected!") - return False - - print_success(f"All {len(component_ids)} component IDs are unique") - - # Verify namespacing pattern - for comp_id, comp_data in components.items(): - file_path = comp_data.get("file_path", "") - print_info(f"Component ID: {comp_id}") - print_info(f" File: {file_path}") - - # Component ID should include path information to prevent collisions - if "main" in file_path and "main" not in comp_id: - print_warning(f" Component ID may not include path namespace") - elif "deps" in file_path and "deps" not in comp_id: - print_warning(f" Component ID may not include path namespace") - - print_success("Component namespacing validated") - return True - - except Exception as e: - print_error(f"Test failed: {str(e)}") - import traceback - traceback.print_exc() - return False - finally: - shutil.rmtree(output_dir, ignore_errors=True) - - -def test_cross_path_dependencies(): - """Test 4: Dependencies between components in different paths.""" - print_header("Test 4: Cross-Path Dependencies") - - main_path = Path(__file__).parent / "main" - deps_path = Path(__file__).parent / "deps" - output_dir = tempfile.mkdtemp(prefix="codewiki_test_deps_") - - try: - config = create_test_config( - str(main_path), - output_dir, - additional_paths=[str(deps_path)] - ) - - print_info("Note: service.py imports from deps/helper.py") - builder = DependencyGraphBuilder(config) - components_dict, leaf_nodes = builder.build_dependency_graph() - - # components_dict is a Dict[str, Node], convert to dict for testing - components = {comp_id: comp.model_dump() for comp_id, comp in components_dict.items()} - - # Build edges from component dependencies - edges = [] - for comp_id, comp_data in components.items(): - depends_on = comp_data.get("depends_on", []) - if isinstance(depends_on, set): - depends_on = list(depends_on) - for dep in depends_on: - edges.append({"source": comp_id, "target": dep}) - - print_success(f"Found {len(components)} components") - print_success(f"Found {len(edges)} dependency edges") - - # Look for cross-path dependencies - cross_path_deps = [] - for edge in edges: - source_id = edge.get("source") - target_id = edge.get("target") - - if source_id and target_id: - source_comp = components.get(source_id, {}) - target_comp = components.get(target_id, {}) - - source_path = source_comp.get("file_path", "") - target_path = target_comp.get("file_path", "") - - # Check if dependency crosses path boundaries - source_in_main = "main" in source_path - source_in_deps = "deps" in source_path - target_in_main = "main" in target_path - target_in_deps = "deps" in target_path - - if (source_in_main and target_in_deps) or (source_in_deps and target_in_main): - cross_path_deps.append((source_path, target_path)) - print_info(f"Cross-path dependency: {os.path.basename(source_path)} → {os.path.basename(target_path)}") - - if cross_path_deps: - print_success(f"Found {len(cross_path_deps)} cross-path dependencies") - else: - print_warning("No cross-path dependencies detected (expected at least 1)") - print_warning("service.py should depend on helper.py") - - return True - - except Exception as e: - print_error(f"Test failed: {str(e)}") - import traceback - traceback.print_exc() - return False - finally: - shutil.rmtree(output_dir, ignore_errors=True) - - -def test_invalid_path_handling(): - """Test 6: Invalid path handling.""" - print_header("Test 6: Invalid Path Handling") - - main_path = Path(__file__).parent / "main" - invalid_path = "/nonexistent/path/that/does/not/exist" - output_dir = tempfile.mkdtemp(prefix="codewiki_test_invalid_") - - try: - config = create_test_config( - str(main_path), - output_dir, - additional_paths=[invalid_path] - ) - - print_info(f"Testing with invalid path: {invalid_path}") - - try: - # This should raise an error during validation - config.validate_source_paths() - print_error("Expected validation error for invalid path, but none was raised") - return False - - except (ValueError, OSError) as e: - print_success(f"Correctly raised error for invalid path: {str(e)}") - return True - - except Exception as e: - print_error(f"Unexpected error: {str(e)}") - return False - finally: - shutil.rmtree(output_dir, ignore_errors=True) - - -def test_empty_additional_paths(): - """Test 7: Empty additional paths list.""" - print_header("Test 7: Empty Additional Paths") - - main_path = Path(__file__).parent / "main" - output_dir = tempfile.mkdtemp(prefix="codewiki_test_empty_") - - try: - # Test with empty list - config = create_test_config( - str(main_path), - output_dir, - additional_paths=[] - ) - - print_info("Testing with empty additional_paths list") - builder = DependencyGraphBuilder(config) - components_dict, leaf_nodes = builder.build_dependency_graph() - - # components_dict is a Dict[str, Node], convert to dict for testing - components = {comp_id: comp.model_dump() for comp_id, comp in components_dict.items()} - if components: - print_success(f"Found {len(components)} components from main path") - return True - else: - print_error("No components found") - return False - - except Exception as e: - print_error(f"Test failed: {str(e)}") - import traceback - traceback.print_exc() - return False - finally: - shutil.rmtree(output_dir, ignore_errors=True) - - -def test_relative_vs_absolute_paths(): - """Test 8: Relative vs absolute path handling.""" - print_header("Test 8: Relative vs Absolute Paths") - - main_path = Path(__file__).parent / "main" - deps_path_relative = "deps" # Relative to main - deps_path_absolute = str(Path(__file__).parent / "deps") # Absolute - output_dir = tempfile.mkdtemp(prefix="codewiki_test_paths_") - - try: - print_info("Testing with absolute path") - config_abs = create_test_config( - str(main_path), - output_dir, - additional_paths=[deps_path_absolute] - ) - - builder = DependencyGraphBuilder(config_abs) - components_dict, leaf_nodes = builder.build_dependency_graph() - - # components_dict is a Dict[str, Node], convert to dict for testing - components_abs = {comp_id: comp.model_dump() for comp_id, comp in components_dict.items()} - - print_success(f"Absolute path: Found {len(components_abs)} components") - - # Note: Relative paths may not work as expected without proper resolution - print_info("\nNote: Relative paths require proper resolution logic") - print_info("Expected behavior: System should resolve relative to main repo_path") - - if len(components_abs) > 2: # Should have main + deps components - print_success("Multi-path analysis working with absolute paths") - return True - else: - print_warning("May only have analyzed main path") - return True # Not a failure, just informational - - except Exception as e: - print_error(f"Test failed: {str(e)}") - import traceback - traceback.print_exc() - return False - finally: - shutil.rmtree(output_dir, ignore_errors=True) - - -def run_all_tests(): - """Run all multi-path tests.""" - print_header("CodeWiki Multi-Path Feature Test Suite") - - tests = [ - ("Single Path (Backward Compatibility)", test_single_path), - ("Multiple Paths With Unique Components", test_multiple_paths), - ("Component ID Namespacing", test_component_namespacing), - ("Cross-Path Dependencies", test_cross_path_dependencies), - ("Invalid Path Handling", test_invalid_path_handling), - ("Empty Additional Paths", test_empty_additional_paths), - ("Relative vs Absolute Paths", test_relative_vs_absolute_paths), - ] - - test_results = TestResults() - - for test_name, test_func in tests: - try: - result = test_func() - test_results.add_test(test_name, result) - except Exception as e: - print_error(f"Test '{test_name}' crashed: {str(e)}") - import traceback - traceback.print_exc() - test_results.add_test(test_name, False) - - return test_results.print_summary() - - -if __name__ == "__main__": - sys.exit(run_all_tests()) diff --git a/test_short_id_normalization.py b/test_short_id_normalization.py index cdd71f0a..3d93dbdc 100644 --- a/test_short_id_normalization.py +++ b/test_short_id_normalization.py @@ -58,8 +58,7 @@ def test_short_id_normalization(): namespace="deps/lib-auth", component_type="class", file_path="/path/to/handlers.py", - relative_path="auth/handlers.py", - is_from_deps=True + relative_path="auth/handlers.py" ), } diff --git a/verify_fqdn_implementation.py b/verify_fqdn_implementation.py index 262c1062..9ea304f2 100755 --- a/verify_fqdn_implementation.py +++ b/verify_fqdn_implementation.py @@ -12,7 +12,7 @@ def check_node_model(): with open('codewiki/src/be/dependency_analyzer/models/core.py', 'r') as f: content = f.read() - required_fields = ['short_id:', 'namespace:', 'is_from_deps:'] + required_fields = ['short_id:', 'namespace:'] checks = {field: field in content for field in required_fields} print("✅ Node Model FQDN Fields Check:") @@ -28,14 +28,12 @@ def check_ast_parser(): content = f.read() checks = { - 'FQDN comment in multi-path': '# FQDN metadata fields' in content, + 'FQDN metadata comment': '# FQDN metadata fields' in content, 'fqdn = f"{namespace}.{original_id}"': 'fqdn = f"{namespace}.{original_id}"' in content, 'components[fqdn] = node': 'components[fqdn] = node' in content, 'id=fqdn': 'id=fqdn' in content, 'short_id=original_id': 'short_id=original_id' in content, 'namespace=namespace': 'namespace=namespace' in content, - 'is_from_deps=(repo_index > 0)': 'is_from_deps=(repo_index > 0)' in content, - 'is_from_deps=False': 'is_from_deps=False' in content, } print("\n✅ AST Parser FQDN Implementation Check:") From ca84ff805808ecc4ac2655755fd9fe95b6cfa359 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Thu, 12 Nov 2026 22:43:09 -0500 Subject: [PATCH 02/12] feat(codewiki): CU-17tkuw66ma4 log each answered code-graph tool call Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/agent_tools/code_graph.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/codewiki/src/be/agent_tools/code_graph.py b/codewiki/src/be/agent_tools/code_graph.py index f9085210..f5d33ad5 100644 --- a/codewiki/src/be/agent_tools/code_graph.py +++ b/codewiki/src/be/agent_tools/code_graph.py @@ -22,6 +22,7 @@ from __future__ import annotations +import json import logging import os from typing import Any, Dict, List, Optional @@ -123,6 +124,8 @@ async def call_hub_tool(**args: Any) -> str: content = body.get("content") if isinstance(body, dict) else None if not isinstance(content, str): raise ToolFailed(f"{name} failed: the hub's answer had no content") + # One line per answered call, so a run's log shows what the writers looked up. + logger.info(f"Code-graph tool {name} answered ({len(content)} chars) for {json.dumps(args, sort_keys=True)[:200]}") return content return Tool.from_schema( From a3394e82d0d5fa85300db60ec8c1d63e341052eb Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Thu, 12 Nov 2026 23:58:11 -0500 Subject: [PATCH 03/12] feat(codewiki): CU-17tkuw66ma4 record tokens per answering model and merge them into the run's usage file Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/agent_orchestrator.py | 2 + .../generate_sub_module_documentations.py | 2 + codewiki/src/be/llm_services.py | 2 + codewiki/src/be/usage_tracker.py | 113 ++++++++++++++++++ 4 files changed, 119 insertions(+) create mode 100644 codewiki/src/be/usage_tracker.py diff --git a/codewiki/src/be/agent_orchestrator.py b/codewiki/src/be/agent_orchestrator.py index d3df8122..52fbbadc 100644 --- a/codewiki/src/be/agent_orchestrator.py +++ b/codewiki/src/be/agent_orchestrator.py @@ -10,6 +10,7 @@ from pydantic_ai import Agent from pydantic_ai.usage import UsageLimits +from codewiki.src.be.usage_tracker import record_agent_run # import logfire import logging import os @@ -192,6 +193,7 @@ async def process_module(self, module_name: str, components: Dict[str, Node], deps=deps, usage_limits=UsageLimits(request_limit=1000), ) + record_agent_run(module_name, result) # Save updated module tree file_manager.save_json(deps.module_tree, module_tree_path) diff --git a/codewiki/src/be/agent_tools/generate_sub_module_documentations.py b/codewiki/src/be/agent_tools/generate_sub_module_documentations.py index 41b9598a..05a879f6 100644 --- a/codewiki/src/be/agent_tools/generate_sub_module_documentations.py +++ b/codewiki/src/be/agent_tools/generate_sub_module_documentations.py @@ -18,6 +18,7 @@ from pydantic_ai import RunContext, Tool, Agent from pydantic_ai.usage import UsageLimits +from codewiki.src.be.usage_tracker import record_agent_run from codewiki.src.be.agent_tools.deps import CodeWikiDeps from codewiki.src.be.agent_tools.read_code_components import read_code_components_tool @@ -149,6 +150,7 @@ async def generate_sub_module_documentation( deps=ctx.deps, usage_limits=UsageLimits(request_limit=1000), ) + record_agent_run(sub_module_name, result) finally: # remove the sub-module name from the path to current module and the module tree deps.path_to_current_module.pop() diff --git a/codewiki/src/be/llm_services.py b/codewiki/src/be/llm_services.py index 058aba98..441e5f25 100644 --- a/codewiki/src/be/llm_services.py +++ b/codewiki/src/be/llm_services.py @@ -21,6 +21,7 @@ from openai import OpenAI, OpenAIError from codewiki.src.config import Config +from codewiki.src.be.usage_tracker import record_completion # Global request counter for logging progress @@ -446,6 +447,7 @@ def call_llm( logger.info(f" └─ 🚀 Sending request to LLM API...") response = client.chat.completions.create(**kwargs) + record_completion(kwargs.get("model") or model or "unknown", response) response_content = response.choices[0].message.content response_tokens = len(response_content) // 4 # Approximate token count diff --git a/codewiki/src/be/usage_tracker.py b/codewiki/src/be/usage_tracker.py new file mode 100644 index 00000000..a7d598d5 --- /dev/null +++ b/codewiki/src/be/usage_tracker.py @@ -0,0 +1,113 @@ +""" +What a CodeWiki run spent, per model, from the model's own answers. + +Every agent run (the module writers) goes through a fallback model, so the model that answered is read +off each response (`ModelResponse.model_name`), never assumed from the configuration. The clustering +calls go through the OpenAI client directly and are recorded from that response's usage. + +Totals are logged as they grow, and, when `CODEWIKI_USAGE_FILE` is set, merged into that file in the +Code Documentation workflow's usage format (`usage.by_model[model] = {input, output, cacheRead}`, plus a +`funnel.stage2` entry), which the workflow's final report sends to the hub. Input tokens exclude cache +reads, which are counted on their own, as the hub prices them. +""" + +from __future__ import annotations + +import json +import logging +import os +import threading +from typing import Any, Dict + +logger = logging.getLogger(__name__) + +_lock = threading.Lock() +_by_model: Dict[str, Dict[str, int]] = {} +_runs = {"agent_runs": 0, "requests": 0, "tool_calls": 0} + + +def _add(model: str, input_tokens: int, output_tokens: int, cache_read: int) -> None: + entry = _by_model.setdefault(model or "unknown", {"input": 0, "output": 0, "cacheRead": 0}) + entry["input"] += max(0, int(input_tokens or 0)) + entry["output"] += max(0, int(output_tokens or 0)) + entry["cacheRead"] += max(0, int(cache_read or 0)) + + +def record_agent_run(label: str, result: Any) -> None: + """One finished agent run: each of its model responses, attributed to the model that answered it.""" + try: + from pydantic_ai.messages import ModelResponse, ToolCallPart + except ImportError: # pragma: no cover - pydantic_ai is a dependency + return + run_tokens = {"input": 0, "output": 0, "cacheRead": 0, "requests": 0, "tool_calls": 0} + with _lock: + for message in result.all_messages(): + if not isinstance(message, ModelResponse): + continue + usage = message.usage + cache_read = int(getattr(usage, "cache_read_tokens", 0) or 0) + input_tokens = int(getattr(usage, "input_tokens", 0) or 0) - cache_read + output_tokens = int(getattr(usage, "output_tokens", 0) or 0) + _add(message.model_name or "unknown", input_tokens, output_tokens, cache_read) + run_tokens["input"] += max(0, input_tokens) + run_tokens["output"] += output_tokens + run_tokens["cacheRead"] += cache_read + run_tokens["requests"] += 1 + run_tokens["tool_calls"] += sum(1 for part in message.parts if isinstance(part, ToolCallPart)) + _runs["agent_runs"] += 1 + _runs["requests"] += run_tokens["requests"] + _runs["tool_calls"] += run_tokens["tool_calls"] + logger.info( + f"💰 Usage {label}: {run_tokens['requests']} requests, {run_tokens['tool_calls']} tool calls, " + f"{run_tokens['input']} input + {run_tokens['cacheRead']} cache-read + {run_tokens['output']} output tokens; " + f"run total {json.dumps(_by_model, sort_keys=True)}" + ) + _write() + + +def record_completion(model: str, response: Any) -> None: + """One direct chat completion (clustering), from the OpenAI response's usage.""" + usage = getattr(response, "usage", None) + if usage is None: + return + details = getattr(usage, "prompt_tokens_details", None) + cache_read = int(getattr(details, "cached_tokens", 0) or 0) if details is not None else 0 + with _lock: + _add(model, int(getattr(usage, "prompt_tokens", 0) or 0) - cache_read, int(getattr(usage, "completion_tokens", 0) or 0), cache_read) + _runs["requests"] += 1 + logger.info(f"💰 Usage {model}: {getattr(usage, 'prompt_tokens', 0)} prompt + {getattr(usage, 'completion_tokens', 0)} completion tokens") + _write() + + +def totals() -> Dict[str, Any]: + with _lock: + return {"by_model": json.loads(json.dumps(_by_model)), **_runs} + + +_prior: Dict[str, Any] | None = None + + +def _write() -> None: + """Merge this run's totals into `CODEWIKI_USAGE_FILE` (the workflow's usage artifact): what earlier stages wrote, read once, plus this run's.""" + global _prior + path = os.environ.get("CODEWIKI_USAGE_FILE") + if not path: + return + with _lock: + if _prior is None: + try: + with open(path, "r", encoding="utf-8") as handle: + _prior = json.load(handle) + except (OSError, ValueError): + _prior = {} + merged = {model: dict(values) for model, values in (_prior.get("usage", {}).get("by_model") or {}).items()} + for model, values in _by_model.items(): + entry = merged.setdefault(model, {"input": 0, "output": 0, "cacheRead": 0}) + for key in ("input", "output", "cacheRead"): + entry[key] = entry.get(key, 0) + values[key] + funnel = dict(_prior.get("funnel") or {}) + funnel["stage2"] = {"engine": "codewiki", **_runs} + tmp = f"{path}.tmp" + with open(tmp, "w", encoding="utf-8") as handle: + json.dump({"usage": {"by_model": merged}, "funnel": funnel}, handle) + os.replace(tmp, path) From 1d355bd4235f72810da80cc8c1a502d87794b733 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 00:37:11 -0500 Subject: [PATCH 04/12] feat(codewiki): CU-17tkuw66ma4 start every writer from the code graph's ecosystem facts Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/flamingo_guidelines.py | 51 ++++++++++++++++++++++++++ codewiki/src/be/prompt_template.py | 15 ++++++-- 2 files changed, 62 insertions(+), 4 deletions(-) diff --git a/codewiki/src/be/flamingo_guidelines.py b/codewiki/src/be/flamingo_guidelines.py index 97761d68..304e4fac 100644 --- a/codewiki/src/be/flamingo_guidelines.py +++ b/codewiki/src/be/flamingo_guidelines.py @@ -379,3 +379,54 @@ def get_custom_instructions_section() -> str: "\n\n---\n" ) + + +ECOSYSTEM_FACTS_ENV_VAR = "ECOSYSTEM_FACTS_PATH" + + +def load_ecosystem_facts() -> str: + """ + The repository's ecosystem facts, generated by the hub from the code graph (`ecosystem.md`: the + artifacts it publishes and consumes with their pinned versions, who consumes it, its delivery chain). + + The Code Documentation workflow downloads them before this stage ("Fetch ecosystem facts") and names + the file in ECOSYSTEM_FACTS_PATH. The same facts reach every other documentation stage and the code + reviewer inline, so CodeWiki's writers start from them instead of each asking the graph for them again. + Optional: a repository with no graph yet has none. + """ + facts_path = os.environ.get(ECOSYSTEM_FACTS_ENV_VAR) + if not facts_path: + logger.info(f"[CodeWiki] {ECOSYSTEM_FACTS_ENV_VAR} not set - continuing without ecosystem facts") + return "" + try: + path = Path(facts_path) + if not path.exists(): + logger.info(f"[CodeWiki] No ecosystem facts at {facts_path} (no code graph for this repository yet)") + return "" + content = path.read_text(encoding="utf-8").strip() + logger.info(f"[CodeWiki] Loaded ecosystem facts from the code graph ({len(content)} chars)") + return content + except Exception as e: + logger.warning(f"[CodeWiki] Failed to load ecosystem facts: {e}") + return "" + + +ECOSYSTEM_FACTS = load_ecosystem_facts() + + +def get_ecosystem_facts_section() -> str: + """ + The ecosystem facts as a prompt section, or empty when there are none. Verbatim, braces included: the + prompts are filled by exact-placeholder replacement, never `.format()`, so escaping would hand the model + doubled braces in `${VAR}`-style values. + """ + if not ECOSYSTEM_FACTS: + return "" + return ( + "\n" + "Generated from the code graph: the ground truth for which artifacts this repository publishes, " + "which it depends on and at which versions, and which repositories consume it. Never contradict " + "these facts or invent a relationship they do not list.\n\n" + + ECOSYSTEM_FACTS + + "\n\n\n" + ) diff --git a/codewiki/src/be/prompt_template.py b/codewiki/src/be/prompt_template.py index 3fd6b8b8..1ab39727 100644 --- a/codewiki/src/be/prompt_template.py +++ b/codewiki/src/be/prompt_template.py @@ -2,13 +2,17 @@ from codewiki.src.be.flamingo_guidelines import ( get_guidelines_section, get_custom_instructions_section, - get_validation_rules_section + get_validation_rules_section, + get_ecosystem_facts_section, ) # Get sections for prompt injection (empty string if not available) _CUSTOM_INSTRUCTIONS_SECTION = get_custom_instructions_section() _GUIDELINES_SECTION = get_guidelines_section() _VALIDATION_RULES_SECTION = get_validation_rules_section() +# The served code-graph facts about this repository's dependencies (`ecosystem.md`), the ones every other +# documentation stage and the code reviewer get inline. +_ECOSYSTEM_FACTS_SECTION = get_ecosystem_facts_section() # Code from other repositories is NOT parsed or copied: the writing agents look @@ -19,7 +23,8 @@ When code-graph tools are available, look such code up instead of guessing: - `resolve_code_imports`: what an import refers to and which repository defines it - `get_code_symbol`: a definition, its signature and its callers -- `get_repo_ecosystem`: which artifacts and versions this repository uses +- `find_code_symbols`: the definitions under a package, module or crate (`com.openframe.data`, `@scope/pkg`), whichever repository holds them +Which artifacts this repository publishes and consumes, at which versions, and who consumes it, are in ECOSYSTEM_FACTS when present: read them there rather than asking for them again. Document how THIS repository uses that code, never the other repository's internals. Tool results arrive wrapped in a `<< str: - Code-graph tools (when available): look up code this repository imports from other repositories {_CODE_GRAPH_TOOLS_SECTION} +{_ECOSYSTEM_FACTS_SECTION} {{custom_instructions}} """.strip() @@ -572,6 +578,7 @@ def format_module_display_name(module_name: str) -> str: - Code-graph tools (when available): look up code this repository imports from other repositories {_CODE_GRAPH_TOOLS_SECTION} +{_ECOSYSTEM_FACTS_SECTION} {{custom_instructions}} """.strip() @@ -599,7 +606,7 @@ def format_module_display_name(module_name: str) -> str: """.strip() REPO_OVERVIEW_PROMPT = f""" -{_CUSTOM_INSTRUCTIONS_SECTION}{_GUIDELINES_SECTION}{_VALIDATION_RULES_SECTION}You are an AI documentation assistant. Your task is to generate a brief overview of the {{repo_name}} repository. +{_CUSTOM_INSTRUCTIONS_SECTION}{_GUIDELINES_SECTION}{_VALIDATION_RULES_SECTION}{_ECOSYSTEM_FACTS_SECTION}You are an AI documentation assistant. Your task is to generate a brief overview of the {{repo_name}} repository. The overview should be a brief documentation of the repository, including: - The purpose of the repository @@ -633,7 +640,7 @@ def format_module_display_name(module_name: str) -> str: """.strip() MODULE_OVERVIEW_PROMPT = f""" -{_CUSTOM_INSTRUCTIONS_SECTION}{_GUIDELINES_SECTION}{_VALIDATION_RULES_SECTION}You are an AI documentation assistant. Your task is to generate a brief overview of the {{display_name}} module. +{_CUSTOM_INSTRUCTIONS_SECTION}{_GUIDELINES_SECTION}{_VALIDATION_RULES_SECTION}{_ECOSYSTEM_FACTS_SECTION}You are an AI documentation assistant. Your task is to generate a brief overview of the {{display_name}} module. **IMPORTANT - Module Naming:** - **File name**: `{{module_name}}.md` (keep underscores for filename) From ad9613e6c8dc625e2258f2f1b5dac59543412e6e Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 01:00:23 -0500 Subject: [PATCH 05/12] fix(codewiki): CU-17tkuw66ma4 re-read the usage file on every write; one module-level logger in config Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/usage_tracker.py | 31 +++++++++++++++++-------------- codewiki/src/config.py | 9 +++------ 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/codewiki/src/be/usage_tracker.py b/codewiki/src/be/usage_tracker.py index a7d598d5..1b261533 100644 --- a/codewiki/src/be/usage_tracker.py +++ b/codewiki/src/be/usage_tracker.py @@ -84,30 +84,33 @@ def totals() -> Dict[str, Any]: return {"by_model": json.loads(json.dumps(_by_model)), **_runs} -_prior: Dict[str, Any] | None = None +# What this process has already added to the file, per model: each write re-reads the file and replaces THIS +# share only, so an update another writer made in between is kept, never overwritten by a stale copy. +_written: Dict[str, Dict[str, int]] = {} def _write() -> None: - """Merge this run's totals into `CODEWIKI_USAGE_FILE` (the workflow's usage artifact): what earlier stages wrote, read once, plus this run's.""" - global _prior + """Merge this run's totals into `CODEWIKI_USAGE_FILE` (the workflow's usage artifact), re-read on every write.""" + global _written path = os.environ.get("CODEWIKI_USAGE_FILE") if not path: return with _lock: - if _prior is None: - try: - with open(path, "r", encoding="utf-8") as handle: - _prior = json.load(handle) - except (OSError, ValueError): - _prior = {} - merged = {model: dict(values) for model, values in (_prior.get("usage", {}).get("by_model") or {}).items()} + try: + with open(path, "r", encoding="utf-8") as handle: + current = json.load(handle) + except (OSError, ValueError): + current = {} + merged = {model: dict(values) for model, values in (current.get("usage", {}).get("by_model") or {}).items()} for model, values in _by_model.items(): entry = merged.setdefault(model, {"input": 0, "output": 0, "cacheRead": 0}) + before = _written.get(model, {}) for key in ("input", "output", "cacheRead"): - entry[key] = entry.get(key, 0) + values[key] - funnel = dict(_prior.get("funnel") or {}) + entry[key] = max(0, entry.get(key, 0) - before.get(key, 0)) + values[key] + funnel = dict(current.get("funnel") or {}) funnel["stage2"] = {"engine": "codewiki", **_runs} - tmp = f"{path}.tmp" + tmp = f"{path}.{os.getpid()}.tmp" with open(tmp, "w", encoding="utf-8") as handle: - json.dump({"usage": {"by_model": merged}, "funnel": funnel}, handle) + json.dump({**current, "usage": {**(current.get("usage") or {}), "by_model": merged}, "funnel": funnel}, handle) os.replace(tmp, path) + _written = {model: dict(values) for model, values in _by_model.items()} diff --git a/codewiki/src/config.py b/codewiki/src/config.py index f86045dc..6b3312f2 100644 --- a/codewiki/src/config.py +++ b/codewiki/src/config.py @@ -17,10 +17,13 @@ from dataclasses import dataclass, field, fields, asdict from typing import Optional, List, Dict, Any import argparse +import logging import os import sys from dotenv import load_dotenv from codewiki.src.be.flamingo_guidelines import escape_format_braces + +logger = logging.getLogger(__name__) load_dotenv() # Constants @@ -193,9 +196,6 @@ def validate_repo_path(self) -> None: Raises: ValueError: If the path does not exist or is not a directory """ - import logging - logger = logging.getLogger(__name__) - logger.info("🔍 Validating repository path...") if not os.path.exists(self.repo_path): @@ -206,9 +206,6 @@ def validate_repo_path(self) -> None: def get_prompt_addition(self) -> str: """Generate prompt additions based on agent instructions.""" - import logging - logger = logging.getLogger(__name__) - if not self.agent_instructions: logger.debug("📋 get_prompt_addition: No agent_instructions configured") return "" From 484e2546aae8d89b25dbc26fd6b65502d4c55756 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 09:22:28 -0500 Subject: [PATCH 06/12] fix(models): CU-17tkuw66ma4 call OpenAI through Responses and Claude natively with caching; retry passing provider errors gpt-5.6-terra refused every tool call on chat completions, so each agent request fell back to Claude, sent through the OpenAI-compatible endpoint with no prompt caching. The agent model is now chosen by its provider's host: the Responses API on api.openai.com, the native Messages API with cache breakpoints on api.anthropic.com, chat completions elsewhere. SDK calls retry dropped connections, rate limits and 5xx, and a module whose run still fails on such an error is run again after 1 and 5 minutes. Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/agent_orchestrator.py | 49 +++++++- codewiki/src/be/llm_services.py | 171 +++++++++++--------------- 2 files changed, 114 insertions(+), 106 deletions(-) diff --git a/codewiki/src/be/agent_orchestrator.py b/codewiki/src/be/agent_orchestrator.py index 52fbbadc..067c86f1 100644 --- a/codewiki/src/be/agent_orchestrator.py +++ b/codewiki/src/be/agent_orchestrator.py @@ -9,7 +9,16 @@ """ from pydantic_ai import Agent +from pydantic_ai.exceptions import ( + FallbackExceptionGroup, + ModelConnectionError, + ModelRateLimitError, + ModelRequestTimeout, + ModelServerError, + ModelUnavailableError, +) from pydantic_ai.usage import UsageLimits +import asyncio from codewiki.src.be.usage_tracker import record_agent_run # import logfire import logging @@ -21,6 +30,25 @@ logger = logging.getLogger(__name__) +# A module whose agent run fails on a passing provider problem (a dropped connection, a rate limit, an outage) is +# run again, after the SDK's own retries are spent: these waits outlast a short outage or a laptop's network coming +# back. Any other error fails the module at once. +MODULE_RUN_RETRY_DELAYS_S = (60, 300) +_TRANSIENT_MODEL_ERRORS = ( + ModelConnectionError, + ModelRateLimitError, + ModelRequestTimeout, + ModelServerError, + ModelUnavailableError, +) + + +def is_transient_model_error(error: BaseException) -> bool: + """Whether a failed agent run may succeed when run again: every model it tried failed on a passing problem.""" + if isinstance(error, FallbackExceptionGroup): + return all(is_transient_model_error(child) for child in error.exceptions) + return isinstance(error, _TRANSIENT_MODEL_ERRORS) + # try: # # Configure logfire with environment variables for Docker compatibility # logfire_token = os.getenv('LOGFIRE_TOKEN') @@ -188,11 +216,22 @@ async def process_module(self, module_name: str, components: Dict[str, Node], logger.info("") try: - result = await agent.run( - user_prompt, - deps=deps, - usage_limits=UsageLimits(request_limit=1000), - ) + for attempt, delay in enumerate((*MODULE_RUN_RETRY_DELAYS_S, None), start=1): + try: + result = await agent.run( + user_prompt, + deps=deps, + usage_limits=UsageLimits(request_limit=1000), + ) + break + except Exception as run_error: + if delay is None or not is_transient_model_error(run_error): + raise + logger.warning( + f"⏳ {module_name}: attempt {attempt} failed on a passing provider error " + f"({type(run_error).__name__}: {run_error}); running it again in {delay}s" + ) + await asyncio.sleep(delay) record_agent_run(module_name, result) # Save updated module tree diff --git a/codewiki/src/be/llm_services.py b/codewiki/src/be/llm_services.py index 441e5f25..911da85f 100644 --- a/codewiki/src/be/llm_services.py +++ b/codewiki/src/be/llm_services.py @@ -15,8 +15,16 @@ from pydantic_ai.models.openai import OpenAIModel from pydantic_ai.models.openai import OpenAIModelSettings -logger = logging.getLogger(__name__) +import re +from urllib.parse import urlparse + +from pydantic_ai.models import Model +from pydantic_ai.models.anthropic import AnthropicModel, AnthropicModelSettings +from pydantic_ai.models.openai import OpenAIResponsesModel, OpenAIResponsesModelSettings +from pydantic_ai.providers.anthropic import AnthropicProvider from pydantic_ai.providers.openai import OpenAIProvider + +logger = logging.getLogger(__name__) from pydantic_ai.models.fallback import FallbackModel from openai import OpenAI, OpenAIError @@ -86,122 +94,83 @@ def get_model_max_token_field(stage: str = 'generation') -> str: return os.environ.get(env_var, 'max_tokens') -def create_main_model(config: Config) -> OpenAIModel: - """ - Create the main LLM model from configuration. +# Every SDK call retries a dropped connection, a rate limit and a 5xx on its own (backoff by the SDK), before +# the agent run sees an error at all. A run that still fails is retried whole by the orchestrator. +AGENT_MODEL_MAX_RETRIES = 6 - NOTE: Pydantic AI currently hardcodes the 'max_tokens' parameter name in OpenAIModelSettings. - For reasoning models (o3, o3-mini) that require 'max_completion_tokens', the direct API call - in call_llm() uses the correct parameter name. If Pydantic AI models fail with "Unrecognized - request argument supplied: max_tokens", the system will fall back to the direct API call. - """ - # Use per-provider max_tokens - max_tokens = getattr(config, 'main_max_tokens', None) or get_max_output_tokens() - # Check if model supports custom temperature (use per-provider field) - temperature = getattr(config, 'main_temperature', 0.0) - temperature_supported = getattr(config, 'main_temperature_supported', True) - # Build settings dict - only include temperature if model supports it - settings_dict = {'max_tokens': max_tokens} - if temperature_supported: - settings_dict['temperature'] = temperature +def _agent_api(base_url: str) -> str: + """Which wire API an agent model is called through, from its provider's host. - # Build provider with per-provider base_url and optional api_version header - base_url = getattr(config, 'main_base_url', None) + - Anthropic (`api.anthropic.com`): the native Messages API, so the agent's repeated prefix (instructions, tool + definitions, the conversation so far) is read from the prompt cache at a tenth of the input price. Through the + OpenAI-compatible endpoint nothing was cached: 6.8M input tokens and 0 cache reads on one repository. + - OpenAI (`api.openai.com`): the Responses API. Reasoning models refuse function tools on chat completions + ("Function tools with reasoning_effort are not supported ... use /v1/responses"), so every agent request + failed there and fell through to the fallback model. + - Any other host: OpenAI-compatible chat completions, as before. + """ + host = (urlparse(base_url).hostname or '').lower() + if host == 'anthropic.com' or host.endswith('.anthropic.com'): + return 'anthropic' + if host == 'api.openai.com': + return 'responses' + return 'chat' + + +def _create_agent_model(stage: str, model_name: str, config: Config) -> Model: + """One agent model (`main` or `fallback`), on the API its provider's host calls for.""" + max_tokens = getattr(config, f'{stage}_max_tokens', None) or get_max_output_tokens() + temperature = getattr(config, f'{stage}_temperature', 0.0) + temperature_supported = getattr(config, f'{stage}_temperature_supported', True) + base_url = getattr(config, f'{stage}_base_url', None) + api_key = getattr(config, f'{stage}_api_key', None) if not base_url: raise ValueError( - "main_base_url is required in configuration for main/generation model.\n" - f"Model: {config.main_model}\n" - "Please set via CLI: --main-base-url \n" - "Or in config file: main_base_url = ''" + f"{stage}_base_url is required in configuration for the {stage} model.\n" + f"Model: {model_name}\n" + f"Please set via CLI: --{stage}-base-url " ) - - # Prepare default headers for API version (Anthropic models) - default_headers = {} - api_version = getattr(config, 'main_api_version', None) - if api_version: - default_headers['anthropic-version'] = api_version - - # Get per-provider API key - api_key = getattr(config, 'main_api_key', None) if not api_key: raise ValueError( - "main_api_key is required in configuration for main/generation model.\n" - f"Model: {config.main_model}\n" - "Please set via CLI: --main-api-key \n" - "Different AI providers require different API keys." + f"{stage}_api_key is required in configuration for the {stage} model.\n" + f"Model: {model_name}\n" + f"Please set via CLI: --{stage}-api-key " ) - return OpenAIModel( - model_name=config.main_model, - provider=OpenAIProvider( - base_url=base_url, - api_key=api_key, - # NOTE: pydantic-ai's OpenAIProvider takes only base_url, api_key, - # openai_client and http_client - there is no default_headers - # parameter (verified against pydantic-ai 2.40.0), so passing one - # raises TypeError. To send anthropic-version here, build an - # AsyncOpenAI client with default_headers and pass it as - # openai_client=. - ), - settings=OpenAIModelSettings(**settings_dict) - ) - - -def create_fallback_model(config: Config) -> OpenAIModel: - """Create the fallback LLM model from configuration.""" - # Use per-provider max_tokens - max_tokens = getattr(config, 'fallback_max_tokens', None) or get_max_output_tokens() - # Check if model supports custom temperature (use per-provider field) - temperature = getattr(config, 'fallback_temperature', 0.0) - temperature_supported = getattr(config, 'fallback_temperature_supported', True) - - # Build settings dict - only include temperature if model supports it - settings_dict = {'max_tokens': max_tokens} + settings: dict = {'max_tokens': max_tokens, 'max_retries': AGENT_MODEL_MAX_RETRIES} if temperature_supported: - settings_dict['temperature'] = temperature - - # Build provider with per-provider base_url and optional api_version header - base_url = getattr(config, 'fallback_base_url', None) - if not base_url: - raise ValueError( - "fallback_base_url is required in configuration for fallback model.\n" - f"Model: {config.fallback_model}\n" - "Please set via CLI: --fallback-base-url \n" - "Or in config file: fallback_base_url = ''" + settings['temperature'] = temperature + + api = _agent_api(base_url) + logger.info(f"🔌 {stage} model {model_name}: {api} API at {base_url}") + if api == 'anthropic': + # The SDK appends /v1/messages itself, so the configured `.../v1` is the host. + anthropic_base = re.sub(r'/v1/?$', '', base_url.rstrip('/')) + return AnthropicModel( + model_name, + provider=AnthropicProvider(api_key=api_key, base_url=anthropic_base), + settings=AnthropicModelSettings( + **settings, + anthropic_cache_instructions=True, + anthropic_cache_tool_definitions=True, + anthropic_cache=True, + ), ) + provider = OpenAIProvider(base_url=base_url, api_key=api_key) + if api == 'responses': + return OpenAIResponsesModel(model_name, provider=provider, settings=OpenAIResponsesModelSettings(**settings)) + return OpenAIModel(model_name=model_name, provider=provider, settings=OpenAIModelSettings(**settings)) - # Prepare default headers for API version (Anthropic models) - default_headers = {} - api_version = getattr(config, 'fallback_api_version', None) - if api_version: - default_headers['anthropic-version'] = api_version - # Get per-provider API key - api_key = getattr(config, 'fallback_api_key', None) - if not api_key: - raise ValueError( - "fallback_api_key is required in configuration for fallback model.\n" - f"Model: {config.fallback_model}\n" - "Please set via CLI: --fallback-api-key \n" - "Different AI providers require different API keys." - ) +def create_main_model(config: Config) -> Model: + """The main agent model (module documentation).""" + return _create_agent_model('main', config.main_model, config) - return OpenAIModel( - model_name=config.fallback_model, - provider=OpenAIProvider( - base_url=base_url, - api_key=api_key, - # NOTE: pydantic-ai's OpenAIProvider takes only base_url, api_key, - # openai_client and http_client - there is no default_headers - # parameter (verified against pydantic-ai 2.40.0), so passing one - # raises TypeError. To send anthropic-version here, build an - # AsyncOpenAI client with default_headers and pass it as - # openai_client=. - ), - settings=OpenAIModelSettings(**settings_dict) - ) +def create_fallback_model(config: Config) -> Model: + """The model an agent request falls back to when the main model fails.""" + return _create_agent_model('fallback', config.fallback_model, config) def create_cluster_model(config: Config) -> OpenAIModel: From c0503ddfd24bd20646f17b6ef658e62e4eae26a5 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 09:47:27 -0500 Subject: [PATCH 07/12] fix(clustering): CU-17tkuw66ma4 place every component in a module The clustering prompt told the model to leave out components that are not essential, and nothing checked what it left out: openframe-oss-frontend's run grouped 376 of 1,077 components, so two thirds of the repository were never documented. The prompts now ask for every component, and after each clustering call the components still left out are placed in an existing module, by the model in batches and then by the longest shared directory. Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/cluster_modules.py | 129 +++++++++++++++++++++++++++++ codewiki/src/be/prompt_template.py | 8 +- 2 files changed, 133 insertions(+), 4 deletions(-) diff --git a/codewiki/src/be/cluster_modules.py b/codewiki/src/be/cluster_modules.py index 7893cf71..9aa09857 100644 --- a/codewiki/src/be/cluster_modules.py +++ b/codewiki/src/be/cluster_modules.py @@ -331,6 +331,133 @@ def normalize_component_ids_by_lookup( return module_tree +# How many left-out components one placement call names, so its prompt stays small whatever the repository's size. +PLACEMENT_BATCH_SIZE = 400 + + +def _component_dir(fqdn: str, components: Dict[str, Node]) -> List[str]: + """The directory a component's file is in, as path segments.""" + node = components.get(fqdn) + path = getattr(node, 'relative_path', None) or fqdn.split('::')[0] + return [part for part in path.replace('\\', '/').split('/')[:-1] if part] + + +def _shared_prefix(a: List[str], b: List[str]) -> int: + n = 0 + for x, y in zip(a, b): + if x != y: + break + n += 1 + return n + + +def place_by_path(module_tree: Dict[str, Any], left_out: List[str], components: Dict[str, Node]) -> Dict[str, List[str]]: + """Each component goes to the module holding the component whose directory shares the longest path with its own. + + A tie goes to the larger module, then to the module named first. Deterministic and free: the net under the model's + placement, so no component is ever left without a module. + """ + module_dirs = { + name: [_component_dir(fqdn, components) for fqdn in info.get('components', [])] + for name, info in module_tree.items() + } + placed: Dict[str, List[str]] = defaultdict(list) + for fqdn in left_out: + own = _component_dir(fqdn, components) + best = max( + module_dirs, + key=lambda name: ( + max((_shared_prefix(own, d) for d in module_dirs[name]), default=0), + len(module_dirs[name]), + -list(module_dirs).index(name), + ), + ) + placed[best].append(fqdn) + return dict(placed) + + +def _ask_placement( + module_tree: Dict[str, Any], + left_out: List[str], + components: Dict[str, Node], + config: Config, +) -> Dict[str, List[str]]: + """Ask the clustering model which existing module each left-out component belongs in.""" + import json + + modules = [] + for name, info in module_tree.items(): + members = info.get('components', []) + sample = ', '.join(fqdn.split('::')[-1] for fqdn in members[:12]) + modules.append(f"- {name} (path: {info.get('path') or '-'}; {len(members)} components, e.g. {sample})") + placed: Dict[str, List[str]] = defaultdict(list) + for start in range(0, len(left_out), PLACEMENT_BATCH_SIZE): + batch = left_out[start:start + PLACEMENT_BATCH_SIZE] + lines = [] + for idx, fqdn in enumerate(batch): + node = components.get(fqdn) + lines.append(f"{idx}: {fqdn.split('::')[-1]} ({getattr(node, 'relative_path', None) or fqdn.split('::')[0]})") + prompt = ( + "The components of a repository were grouped into these modules:\n" + + "\n".join(modules) + + "\n\nThese components were not placed in any module yet. Each one is an integer ID, its name and its file:\n" + + "\n".join(lines) + + "\n\nPlace EVERY component above in the one existing module it belongs to. Use only the module names listed." + " Answer with a JSON object from module name to the list of bare integer IDs placed in it, wrapped in" + " and , with nothing else." + ) + try: + response = call_llm(prompt, config, model=config.cluster_model) + body = response.split('')[1].split('')[0] + answer = json.loads(body) + except Exception as e: # a failed batch is placed by path below + logger.warning(f" ⚠️ Placement call failed for {len(batch)} components: {type(e).__name__}: {e}") + continue + if not isinstance(answer, dict): + continue + taken = set() + for name, ids in answer.items(): + if name not in module_tree or not isinstance(ids, list): + continue + for raw in ids: + try: + idx = int(raw) + except (TypeError, ValueError): + continue + if 0 <= idx < len(batch) and idx not in taken: + taken.add(idx) + placed[name].append(batch[idx]) + return dict(placed) + + +def place_left_out_components( + module_tree: Dict[str, Any], + leaf_nodes: List[str], + components: Dict[str, Node], + config: Config, +) -> int: + """Put every leaf node the clustering left out into a module: the model's choice first, then by path. + + The documentation covers only the components a module holds, so a component no module holds is never documented + (one repository's clustering placed 376 of its 1,077 components). Returns how many were placed. + """ + held = {fqdn for info in module_tree.values() for fqdn in info.get('components', [])} + left_out = [fqdn for fqdn in dict.fromkeys(leaf_nodes) if fqdn in components and fqdn not in held] + if not left_out or not module_tree: + return 0 + logger.info(f"🧩 {len(left_out)} of {len(left_out) + len(held)} components left out by clustering; placing them") + by_model = _ask_placement(module_tree, left_out, components, config) + for name, fqdns in by_model.items(): + module_tree[name].setdefault('components', []).extend(fqdns) + model_placed = {fqdn for fqdns in by_model.values() for fqdn in fqdns} + rest = [fqdn for fqdn in left_out if fqdn not in model_placed] + by_path = place_by_path(module_tree, rest, components) if rest else {} + for name, fqdns in by_path.items(): + module_tree[name].setdefault('components', []).extend(fqdns) + logger.info(f" └─ placed {len(model_placed)} by the model and {len(rest)} by file path") + return len(left_out) + + def cluster_modules( leaf_nodes: List[str], components: Dict[str, Node], @@ -449,6 +576,8 @@ def cluster_modules( logger.debug(f"Skipping clustering for {current_module_name} because the module tree is too small: {len(module_tree)} modules") return {} + place_left_out_components(module_tree, leaf_nodes, components, config) + if current_module_tree == {}: current_module_tree = module_tree else: diff --git a/codewiki/src/be/prompt_template.py b/codewiki/src/be/prompt_template.py index 1ab39727..3e77537e 100644 --- a/codewiki/src/be/prompt_template.py +++ b/codewiki/src/be/prompt_template.py @@ -679,7 +679,7 @@ def format_module_display_name(module_name: str) -> str: """.strip() CLUSTER_REPO_PROMPT = """ -Here is list of all potential core components of the repository (It's normal that some components are not essential to the repository): +Here is the list of every component of the repository: Each component is identified by an INTEGER ID followed by a human-readable description. @@ -720,7 +720,7 @@ def format_module_display_name(module_name: str) -> str: **Why this matters:** If you return quoted IDs like "0", the system will reject your entire response. -Please group the components into modules such that each group contains closely related components that together form a cohesive module. DO NOT include components that are not essential to the repository. +Please group the components into modules such that each group contains closely related components that together form a cohesive module. Place EVERY component listed above in exactly one module: the documentation covers only the components you place, so a component you leave out is never documented. **MODULE NAMING CONVENTION (REQUIRED):** - Use **kebab-case**: lowercase words separated by hyphens @@ -793,7 +793,7 @@ def format_module_display_name(module_name: str) -> str: {module_tree} -Here is list of all potential core components of the module {module_name} (It's normal that some components are not essential to the module): +Here is the list of every component of the module {module_name}: Each component is identified by an INTEGER ID followed by a human-readable description. @@ -834,7 +834,7 @@ def format_module_display_name(module_name: str) -> str: **Why this matters:** If you return quoted IDs like "0", the system will reject your entire response. -Please group the components into smaller sub-modules such that each group contains closely related components. DO NOT include components that are not essential to the module. +Please group the components into smaller sub-modules such that each group contains closely related components. Place EVERY component listed above in exactly one sub-module: a component you leave out is never documented. **MODULE NAMING CONVENTION (REQUIRED):** - Use **kebab-case**: lowercase words separated by hyphens From c41095ad85846b27d3edd4ae4dd6940062abb0c6 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 10:36:33 -0500 Subject: [PATCH 08/12] fix(codewiki): CU-17tkuw66ma4 answer the review of #134 - Read the code-graph tool list once in a worker thread before any agent is built, never on the event loop. - Placement IDs go through the shared normalizer, which logs each rejected ID; an unknown module name is logged too. - A retried module starts from the state its first attempt started with and does not run sub-modules a failed attempt already finished. - Every agent run goes through run_and_record, which records a failed attempt's tokens before raising, so a retry counts every attempt (failed_attempts in the stage 2 funnel). Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/agent_orchestrator.py | 15 ++++++++-- codewiki/src/be/agent_tools/code_graph.py | 9 ++++++ codewiki/src/be/agent_tools/deps.py | 5 +++- .../generate_sub_module_documentations.py | 12 ++++++-- codewiki/src/be/cluster_modules.py | 20 +++++++------ codewiki/src/be/documentation_generator.py | 4 +++ codewiki/src/be/usage_tracker.py | 29 +++++++++++++++++-- 7 files changed, 75 insertions(+), 19 deletions(-) diff --git a/codewiki/src/be/agent_orchestrator.py b/codewiki/src/be/agent_orchestrator.py index 067c86f1..e6c8ab58 100644 --- a/codewiki/src/be/agent_orchestrator.py +++ b/codewiki/src/be/agent_orchestrator.py @@ -19,7 +19,8 @@ ) from pydantic_ai.usage import UsageLimits import asyncio -from codewiki.src.be.usage_tracker import record_agent_run +import copy +from codewiki.src.be.usage_tracker import run_and_record # import logfire import logging import os @@ -216,9 +217,18 @@ async def process_module(self, module_name: str, components: Dict[str, Node], logger.info("") try: + # A retry starts from the state the first attempt started with. Sub-modules a failed attempt finished stay + # finished (deps.finished_sub_modules), so they are neither written nor paid for twice. + start_state = (copy.deepcopy(deps.module_tree), list(deps.path_to_current_module), deps.current_module_name, deps.current_depth) for attempt, delay in enumerate((*MODULE_RUN_RETRY_DELAYS_S, None), start=1): + if attempt > 1: + deps.module_tree = copy.deepcopy(start_state[0]) + deps.path_to_current_module[:] = start_state[1] + deps.current_module_name, deps.current_depth = start_state[2], start_state[3] try: - result = await agent.run( + result = await run_and_record( + agent, + module_name, user_prompt, deps=deps, usage_limits=UsageLimits(request_limit=1000), @@ -232,7 +242,6 @@ async def process_module(self, module_name: str, components: Dict[str, Node], f"({type(run_error).__name__}: {run_error}); running it again in {delay}s" ) await asyncio.sleep(delay) - record_agent_run(module_name, result) # Save updated module tree file_manager.save_json(deps.module_tree, module_tree_path) diff --git a/codewiki/src/be/agent_tools/code_graph.py b/codewiki/src/be/agent_tools/code_graph.py index f5d33ad5..b9fcfe06 100644 --- a/codewiki/src/be/agent_tools/code_graph.py +++ b/codewiki/src/be/agent_tools/code_graph.py @@ -22,6 +22,7 @@ from __future__ import annotations +import asyncio import json import logging import os @@ -190,6 +191,14 @@ def code_graph_tools() -> List[Tool]: return list(_cached_tools) +async def load_code_graph_tools() -> List[Tool]: + """Read the tool list once, in a worker thread, so the async pipeline never waits on the hub inside its loop. + + The pipeline calls this before it builds any agent; every later `code_graph_tools()` is a cache read. + """ + return await asyncio.to_thread(code_graph_tools) + + def reset_code_graph_tools_cache() -> None: """Forget the tool list read from the hub (tests).""" global _cached_tools diff --git a/codewiki/src/be/agent_tools/deps.py b/codewiki/src/be/agent_tools/deps.py index 6e51b9af..bc134b6e 100644 --- a/codewiki/src/be/agent_tools/deps.py +++ b/codewiki/src/be/agent_tools/deps.py @@ -5,7 +5,7 @@ that is passed to agent tools during the CodeWiki documentation generation pipeline. """ -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Any from codewiki.src.be.dependency_analyzer.models.core import Node from codewiki.src.config import Config @@ -23,3 +23,6 @@ class CodeWikiDeps: current_depth: int config: Config # LLM configuration custom_instructions: str = None + # Sub-modules whose documentation run finished, by their path: a module run retried after a passing provider + # error does not write (and pay for) them again. + finished_sub_modules: set = field(default_factory=set) diff --git a/codewiki/src/be/agent_tools/generate_sub_module_documentations.py b/codewiki/src/be/agent_tools/generate_sub_module_documentations.py index 05a879f6..64d941b9 100644 --- a/codewiki/src/be/agent_tools/generate_sub_module_documentations.py +++ b/codewiki/src/be/agent_tools/generate_sub_module_documentations.py @@ -18,7 +18,7 @@ from pydantic_ai import RunContext, Tool, Agent from pydantic_ai.usage import UsageLimits -from codewiki.src.be.usage_tracker import record_agent_run +from codewiki.src.be.usage_tracker import run_and_record from codewiki.src.be.agent_tools.deps import CodeWikiDeps from codewiki.src.be.agent_tools.read_code_components import read_code_components_tool @@ -100,6 +100,10 @@ async def generate_sub_module_documentation( value[sub_module_name] = {"components": core_component_ids, "children": {}} for sub_module_name, core_component_ids in sub_module_specs.items(): + finished_key = (*deps.path_to_current_module, sub_module_name) + if finished_key in deps.finished_sub_modules: + logger.info(f" ✓ Sub-module {sub_module_name} was documented by an earlier attempt; not run again") + continue # Create visual indentation for nested modules indent = " " * deps.current_depth @@ -140,7 +144,9 @@ async def generate_sub_module_documentation( try: # FLAMINGO_PATCH: Added usage_limits to prevent "request_limit of 50" exceeded errors - result = await sub_agent.run( + await run_and_record( + sub_agent, + sub_module_name, format_user_prompt( module_name=deps.current_module_name, core_component_ids=core_component_ids, @@ -150,7 +156,7 @@ async def generate_sub_module_documentation( deps=ctx.deps, usage_limits=UsageLimits(request_limit=1000), ) - record_agent_run(sub_module_name, result) + deps.finished_sub_modules.add(finished_key) finally: # remove the sub-module name from the path to current module and the module tree deps.path_to_current_module.pop() diff --git a/codewiki/src/be/cluster_modules.py b/codewiki/src/be/cluster_modules.py index 9aa09857..75a228ba 100644 --- a/codewiki/src/be/cluster_modules.py +++ b/codewiki/src/be/cluster_modules.py @@ -415,18 +415,20 @@ def _ask_placement( continue if not isinstance(answer, dict): continue + batch_ids = dict(enumerate(batch)) taken = set() for name, ids in answer.items(): - if name not in module_tree or not isinstance(ids, list): + if name not in module_tree: + logger.warning(f" \u274c Placement named module '{name}', which clustering did not create; its IDs are placed by path") continue - for raw in ids: - try: - idx = int(raw) - except (TypeError, ValueError): - continue - if 0 <= idx < len(batch) and idx not in taken: - taken.add(idx) - placed[name].append(batch[idx]) + if not isinstance(ids, list): + logger.warning(f" \u274c Placement for module '{name}' is not a list of IDs; placed by path") + continue + resolved, _, _ = normalize_component_id_list(ids, batch_ids, context=f"placement into '{name}'") + for fqdn in resolved: + if fqdn not in taken: + taken.add(fqdn) + placed[name].append(fqdn) return dict(placed) diff --git a/codewiki/src/be/documentation_generator.py b/codewiki/src/be/documentation_generator.py index ae3c7634..08875cb0 100644 --- a/codewiki/src/be/documentation_generator.py +++ b/codewiki/src/be/documentation_generator.py @@ -17,6 +17,7 @@ logger = logging.getLogger(__name__) # Local imports +from codewiki.src.be.agent_tools.code_graph import load_code_graph_tools from codewiki.src.be.dependency_analyzer import DependencyGraphBuilder from codewiki.src.be.llm_services import call_llm from codewiki.src.be.prompt_template import ( @@ -176,6 +177,9 @@ async def generate_module_documentation(self, components: Dict[str, Any], leaf_n """Generate documentation for all modules using dynamic programming approach.""" logger.info("📝 Stage 3.1: Module Documentation Generation") + # The code-graph tool list, read once off the event loop before any agent is built. + await load_code_graph_tools() + # Prepare output directory working_dir = os.path.abspath(self.config.docs_dir) file_manager.ensure_directory(working_dir) diff --git a/codewiki/src/be/usage_tracker.py b/codewiki/src/be/usage_tracker.py index 1b261533..a549e167 100644 --- a/codewiki/src/be/usage_tracker.py +++ b/codewiki/src/be/usage_tracker.py @@ -23,7 +23,7 @@ _lock = threading.Lock() _by_model: Dict[str, Dict[str, int]] = {} -_runs = {"agent_runs": 0, "requests": 0, "tool_calls": 0} +_runs = {"agent_runs": 0, "failed_attempts": 0, "requests": 0, "tool_calls": 0} def _add(model: str, input_tokens: int, output_tokens: int, cache_read: int) -> None: @@ -35,13 +35,36 @@ def _add(model: str, input_tokens: int, output_tokens: int, cache_read: int) -> def record_agent_run(label: str, result: Any) -> None: """One finished agent run: each of its model responses, attributed to the model that answered it.""" + record_messages(label, result.all_messages()) + + +async def run_and_record(agent: Any, label: str, *args: Any, **kwargs: Any) -> Any: + """Run an agent and record what it spent, a run that fails included. + + A failed run's model responses (answered before the error) were billed too, so they are recorded before the + error is raised: a run that is retried then counts every attempt. + """ + from pydantic_ai import capture_run_messages + + with capture_run_messages() as messages: + try: + result = await agent.run(*args, **kwargs) + except BaseException: + record_messages(f"{label} (failed attempt)", messages, finished=False) + raise + record_agent_run(label, result) + return result + + +def record_messages(label: str, messages: Any, finished: bool = True) -> None: + """Each model response of one agent run, attributed to the model that answered it.""" try: from pydantic_ai.messages import ModelResponse, ToolCallPart except ImportError: # pragma: no cover - pydantic_ai is a dependency return run_tokens = {"input": 0, "output": 0, "cacheRead": 0, "requests": 0, "tool_calls": 0} with _lock: - for message in result.all_messages(): + for message in messages: if not isinstance(message, ModelResponse): continue usage = message.usage @@ -54,7 +77,7 @@ def record_agent_run(label: str, result: Any) -> None: run_tokens["cacheRead"] += cache_read run_tokens["requests"] += 1 run_tokens["tool_calls"] += sum(1 for part in message.parts if isinstance(part, ToolCallPart)) - _runs["agent_runs"] += 1 + _runs["agent_runs" if finished else "failed_attempts"] += 1 _runs["requests"] += run_tokens["requests"] _runs["tool_calls"] += run_tokens["tool_calls"] logger.info( From cff348e1e1263b27da7f0f83d5b054e2d3483963 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 13:52:18 -0500 Subject: [PATCH 09/12] fix(analysis): CU-17tkuw66ma4 leave copied libraries and minified builds out of what is documented A repository that ships its front end's libraries beside its own scripts had them clustered and documented as its own modules once every component was placed: on meshcentral most of the 143 modules, and both failures (a Chart.js tree of repeated names, and Font Awesome's 1.2M-token file no model can read). The file walk now leaves out, by FORM and LOCATION and never by a list of library names: - minified .js/.css files (average line over 110 characters, GitHub Linguist's rule); - files that open with a /*! library banner; - vendored folders (vendor, third_party, bower_components, ...). It logs what it left out and why. On meshcentral: 69 files and the vendored folder, 13 MB of 21.7 MB of script and style; on ten other repositories: nothing. Co-Authored-By: Claude Opus 5.5 --- .../analysis/analysis_service.py | 12 +++++ .../analysis/repo_analyzer.py | 7 +++ .../be/dependency_analyzer/utils/patterns.py | 9 +++- .../dependency_analyzer/utils/third_party.py | 47 +++++++++++++++++++ 4 files changed, 74 insertions(+), 1 deletion(-) create mode 100644 codewiki/src/be/dependency_analyzer/utils/third_party.py diff --git a/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py b/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py index cdc2d0da..cf9d028b 100644 --- a/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py +++ b/codewiki/src/be/dependency_analyzer/analysis/analysis_service.py @@ -245,6 +245,18 @@ def _analyze_structure( logger.debug("└─ Initializing RepoAnalyzer...") repo_analyzer = RepoAnalyzer(include_patterns, exclude_patterns) result = repo_analyzer.analyze_repository_structure(repo_dir) + skipped = repo_analyzer.third_party_files + if skipped: + by_reason: Dict[str, int] = {} + for reason in skipped.values(): + by_reason[reason] = by_reason.get(reason, 0) + 1 + logger.info( + f"🧾 Left out {len(skipped)} file(s) as someone else's code (" + + ", ".join(f"{n} {reason}" for reason, n in sorted(by_reason.items())) + + "): " + + ", ".join(sorted(skipped)[:20]) + + (f" and {len(skipped) - 20} more" if len(skipped) > 20 else "") + ) logger.debug(f" ├─ Files found: {result.get('summary', {}).get('total_files', 0)}") logger.debug(f" └─ Languages detected: {', '.join(result.get('summary', {}).get('languages', []))}") diff --git a/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py b/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py index e3d42a32..63678759 100644 --- a/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py +++ b/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py @@ -11,6 +11,7 @@ from pathlib import Path from typing import Dict, List, Optional, Union from codewiki.src.be.dependency_analyzer.utils.patterns import DEFAULT_IGNORE_PATTERNS, DEFAULT_INCLUDE_PATTERNS +from codewiki.src.be.dependency_analyzer.utils.third_party import third_party_reason class RepoAnalyzer: @@ -31,6 +32,8 @@ def __init__( ) # Track namespaces when analyzing multiple paths self._namespace_roots: Dict[str, str] = {} # namespace -> repo_path + # Files left out as someone else's code (copied libraries, minified builds): path -> why. + self.third_party_files: Dict[str, str] = {} def analyze_repository_structure(self, repo_dir: Union[str, List[str]]) -> Dict: """ @@ -165,6 +168,10 @@ def build_tree(path: Path, base_path: Path) -> Optional[Dict]: if path.is_file(): if not self._should_include_file(relative_path_str, path.name): return None + reason = third_party_reason(path) + if reason: + self.third_party_files[relative_path_str] = reason + return None size = path.stat().st_size return { diff --git a/codewiki/src/be/dependency_analyzer/utils/patterns.py b/codewiki/src/be/dependency_analyzer/utils/patterns.py index 9fb003f7..defec38f 100644 --- a/codewiki/src/be/dependency_analyzer/utils/patterns.py +++ b/codewiki/src/be/dependency_analyzer/utils/patterns.py @@ -119,9 +119,16 @@ # Documentation ".docusaurus", # Other common patterns - ## Minified files + ## Minified files (any other minified file is found by its form: utils/third_party.py) "*.min.js", "*.min.css", + ## Folders that hold other projects' code (utils/third_party.py's THIRD_PARTY_DIRECTORIES) + "vendor", + "vendors", + "third_party", + "third-party", + "thirdparty", + "bower_components", ## Source maps "*.map", ## Terraform diff --git a/codewiki/src/be/dependency_analyzer/utils/third_party.py b/codewiki/src/be/dependency_analyzer/utils/third_party.py new file mode 100644 index 00000000..1de6e9d0 --- /dev/null +++ b/codewiki/src/be/dependency_analyzer/utils/third_party.py @@ -0,0 +1,47 @@ +""" +Which files are someone else's code: copied libraries and build output, never the repository's own source. + +A repository that ships its front end's libraries beside its own scripts (jQuery, Bootstrap, Chart.js, Font +Awesome, a minified copy of each) had them clustered and documented as its own modules: most of one run's 143 +modules, and its only failures (a 1.2M-token library no model can read). The rules are about a file's FORM and +LOCATION, never a list of library names: + +- minified: a `.js` / `.css` file whose average line is longer than 110 characters, GitHub Linguist's rule for + minified files (a person's code averages well under that; one repository's own files averaged 21 to 78); +- a library banner: the file opens with a `/*!` comment, the form minifiers and library builds keep for the + library's license (`/*! jQuery v3.7.1`, `/*! Bootstrap v5.3.3`); a project's own files open with `/**` or `//`; +- a vendored folder (`vendor`, `third_party`, `bower_components`, …): matched by the walk's own path patterns. + +A file the rules cannot read is kept: leaving out code the repository wrote is worse than documenting a library. +""" +from __future__ import annotations + +from pathlib import Path +from typing import Optional + +MINIFIED_EXTENSIONS = {'.js', '.mjs', '.cjs', '.css'} +# GitHub Linguist's threshold for a minified file (average characters per line). +MINIFIED_AVERAGE_LINE_LENGTH = 110 +# What is read of a file to judge it: enough for any banner and a representative average. +SAMPLE_BYTES = 256 * 1024 +# Folders that hold other projects' code by convention (Linguist's vendored directories). +THIRD_PARTY_DIRECTORIES = ('vendor', 'vendors', 'third_party', 'third-party', 'thirdparty', 'bower_components') + + +def third_party_reason(path: Path) -> Optional[str]: + """Why a file is someone else's code (`minified`, `library banner`), or None for the repository's own.""" + suffix = path.suffix.lower() + if suffix not in MINIFIED_EXTENSIONS: + return None + try: + with open(path, 'rb') as handle: + sample = handle.read(SAMPLE_BYTES) + except OSError: + return None + text = sample.decode('utf-8', errors='replace').lstrip('') + if text.lstrip().startswith('/*!'): + return 'library banner' + lines = text.split('\n') + if len(text) / max(1, len(lines)) > MINIFIED_AVERAGE_LINE_LENGTH: + return 'minified' + return None From 982b63dd39da6af59c3f53fb588984b805aaefb2 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 16:15:48 -0500 Subject: [PATCH 10/12] fix(usage): CU-17tkuw66ma4 a usage file that cannot be merged or written never stops a run Writing the usage file could raise out of every model call (a read-only path, a full disk, a file that is not the expected object) and abort documentation for a report. Any such failure is now logged and the run carries on. Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/usage_tracker.py | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/codewiki/src/be/usage_tracker.py b/codewiki/src/be/usage_tracker.py index a549e167..31d545b9 100644 --- a/codewiki/src/be/usage_tracker.py +++ b/codewiki/src/be/usage_tracker.py @@ -124,16 +124,24 @@ def _write() -> None: current = json.load(handle) except (OSError, ValueError): current = {} - merged = {model: dict(values) for model, values in (current.get("usage", {}).get("by_model") or {}).items()} - for model, values in _by_model.items(): - entry = merged.setdefault(model, {"input": 0, "output": 0, "cacheRead": 0}) - before = _written.get(model, {}) - for key in ("input", "output", "cacheRead"): - entry[key] = max(0, entry.get(key, 0) - before.get(key, 0)) + values[key] - funnel = dict(current.get("funnel") or {}) - funnel["stage2"] = {"engine": "codewiki", **_runs} - tmp = f"{path}.{os.getpid()}.tmp" - with open(tmp, "w", encoding="utf-8") as handle: - json.dump({**current, "usage": {**(current.get("usage") or {}), "by_model": merged}, "funnel": funnel}, handle) - os.replace(tmp, path) + # Usage is a report, never a reason to stop documenting: any failure to merge or write it (a read-only + # path, a full disk, a file that is not the expected object) is logged and the run carries on. + try: + if not isinstance(current, dict): + current = {} + merged = {model: dict(values) for model, values in ((current.get("usage") or {}).get("by_model") or {}).items()} + for model, values in _by_model.items(): + entry = merged.setdefault(model, {"input": 0, "output": 0, "cacheRead": 0}) + before = _written.get(model, {}) + for key in ("input", "output", "cacheRead"): + entry[key] = max(0, entry.get(key, 0) - before.get(key, 0)) + values[key] + funnel = dict(current.get("funnel") or {}) + funnel["stage2"] = {"engine": "codewiki", **_runs} + tmp = f"{path}.{os.getpid()}.tmp" + with open(tmp, "w", encoding="utf-8") as handle: + json.dump({**current, "usage": {**(current.get("usage") or {}), "by_model": merged}, "funnel": funnel}, handle) + os.replace(tmp, path) + except Exception as error: # noqa: BLE001 - see above + logger.warning(f"⚠️ Usage not written to {path}: {type(error).__name__}: {error}") + return _written = {model: dict(values) for model, values in _by_model.items()} From 3842fd5f7ce9a16dbd836229b977335bfe534252 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 16:49:28 -0500 Subject: [PATCH 11/12] fix(analysis,cli): CU-17tkuw66ma4 keep hand-written files with one long line; refuse --additional-paths with its reason - A file averaging over 110 characters a line is minified only if it also has fewer than 200 ordinary lines (non-empty, at most 110 characters), counted over the whole file: meshcentral's agents/meshcmd.js (2,673 ordinary lines plus one 268,000-character data line) is kept, every library still left out (the most any had was 33). - --additional-paths is refused with a message naming what replaced it (the hub's code-graph tools), instead of click's bare "no such option"; nothing is accepted or ignored. CHANGELOG.md records the breaking removal. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 20 ++++++++++++++++++ codewiki/cli/commands/generate.py | 17 +++++++++++++++ .../dependency_analyzer/utils/third_party.py | 21 +++++++++++++++---- 3 files changed, 54 insertions(+), 4 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..d6241925 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,20 @@ +# Changelog + +## Unreleased + +### Breaking changes + +- **`--additional-paths` and `additional_source_paths` are removed.** CodeWiki documents only the repository it + runs in. Other repositories are read through the Flamingo hub's code-graph tools (`HUB_BASE_URL`, + `WEBHOOK_SECRET`, `GITHUB_REPOSITORY`), never copied beside the checkout. Passing `--additional-paths` now stops + with an error saying so; drop the option. + +### Changes + +- Agent models are called by their provider's own API: the OpenAI Responses API on `api.openai.com`, the native + Anthropic API (with prompt caching) on `api.anthropic.com`, chat completions on any other host. +- Passing provider errors (dropped connections, rate limits, outages) are retried, by the SDK and then per module. +- Every component is placed in a module; clustering no longer leaves "non-essential" components undocumented. +- Copied libraries and minified builds are left out of what is documented (minified by form, a `/*!` library + banner, vendored folders), and each run logs what it left out and why. +- Each run's model usage, per answering model and failed attempts included, is written to `CODEWIKI_USAGE_FILE`. diff --git a/codewiki/cli/commands/generate.py b/codewiki/cli/commands/generate.py index 76903e63..41562659 100644 --- a/codewiki/cli/commands/generate.py +++ b/codewiki/cli/commands/generate.py @@ -39,6 +39,15 @@ def parse_patterns(patterns_str: str) -> List[str]: return [p.strip() for p in patterns_str.split(',') if p.strip()] +def _removed_additional_paths(value): + """`--additional-paths` was removed: say so and why, rather than click's bare "no such option".""" + if value: + raise click.UsageError( + "--additional-paths was removed: CodeWiki documents only the repository it runs in, and reads other " + "repositories through the hub's code-graph tools (HUB_BASE_URL, WEBHOOK_SECRET). Drop the option." + ) + + @click.command(name="generate") @click.option( "--output", @@ -138,6 +147,14 @@ def parse_patterns(patterns_str: str) -> List[str]: default=None, help="Maximum depth for hierarchical decomposition (overrides config)", ) +@click.option( + "--additional-paths", + hidden=True, + expose_value=False, + is_eager=True, + callback=lambda ctx, param, value: _removed_additional_paths(value), + help="Removed: other repositories are read through the hub's code-graph tools.", +) @click.pass_context def generate_command( ctx, diff --git a/codewiki/src/be/dependency_analyzer/utils/third_party.py b/codewiki/src/be/dependency_analyzer/utils/third_party.py index 1de6e9d0..a495123d 100644 --- a/codewiki/src/be/dependency_analyzer/utils/third_party.py +++ b/codewiki/src/be/dependency_analyzer/utils/third_party.py @@ -7,7 +7,10 @@ LOCATION, never a list of library names: - minified: a `.js` / `.css` file whose average line is longer than 110 characters, GitHub Linguist's rule for - minified files (a person's code averages well under that; one repository's own files averaged 21 to 78); + minified files (a person's code averages well under that; one repository's own files averaged 21 to 78), unless + it also holds at least 200 ordinary lines (non-empty, at most 110 characters): a hand-written file with one long + embedded data line (meshcentral's agents/meshcmd.js: 2,673 ordinary lines) is code, while minified libraries + have a few dozen at most (OpenLayers 33, each Bootstrap theme 11); - a library banner: the file opens with a `/*!` comment, the form minifiers and library builds keep for the library's license (`/*! jQuery v3.7.1`, `/*! Bootstrap v5.3.3`); a project's own files open with `/**` or `//`; - a vendored folder (`vendor`, `third_party`, `bower_components`, …): matched by the walk's own path patterns. @@ -22,6 +25,10 @@ MINIFIED_EXTENSIONS = {'.js', '.mjs', '.cjs', '.css'} # GitHub Linguist's threshold for a minified file (average characters per line). MINIFIED_AVERAGE_LINE_LENGTH = 110 +# A file this many ordinary lines (non-empty, at most the threshold above) is hand-written whatever its average. +HAND_WRITTEN_MIN_LINES = 200 +# How much of a file the ordinary lines are counted in (a long embedded line can fill the first sample alone). +COUNT_BYTES = 16 * 1024 * 1024 # What is read of a file to judge it: enough for any banner and a representative average. SAMPLE_BYTES = 256 * 1024 # Folders that hold other projects' code by convention (Linguist's vendored directories). @@ -42,6 +49,12 @@ def third_party_reason(path: Path) -> Optional[str]: if text.lstrip().startswith('/*!'): return 'library banner' lines = text.split('\n') - if len(text) / max(1, len(lines)) > MINIFIED_AVERAGE_LINE_LENGTH: - return 'minified' - return None + if len(text) / max(1, len(lines)) <= MINIFIED_AVERAGE_LINE_LENGTH: + return None + try: + with open(path, 'rb') as handle: + whole = handle.read(COUNT_BYTES).decode('utf-8', errors='replace') + except OSError: + return None + ordinary = sum(1 for line in whole.split('\n') if line.strip() and len(line) <= MINIFIED_AVERAGE_LINE_LENGTH) + return None if ordinary >= HAND_WRITTEN_MIN_LINES else 'minified' From 3e5fe70adb4a7968eecacc2bf15533a200bb1920 Mon Sep 17 00:00:00 2001 From: Michael Assraf Date: Fri, 13 Nov 2026 22:03:38 -0500 Subject: [PATCH 12/12] chore(codewiki): CU-17tkuw66ma4 no repository names in examples, comments or prompts Co-Authored-By: Claude Opus 5.5 --- codewiki/src/be/cluster_modules.py | 2 +- codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py | 2 +- codewiki/src/be/dependency_analyzer/ast_parser.py | 2 +- codewiki/src/be/dependency_analyzer/utils/third_party.py | 4 ++-- codewiki/src/be/prompt_template.py | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/codewiki/src/be/cluster_modules.py b/codewiki/src/be/cluster_modules.py index 75a228ba..b6755a60 100644 --- a/codewiki/src/be/cluster_modules.py +++ b/codewiki/src/be/cluster_modules.py @@ -67,7 +67,7 @@ def extract_package_hint(fqdn: str) -> str: Extract package/category from FQDN for human readability. Examples: - "...src.main.java.com.openframe.api.controller.Class" → "controller" + "...src.main.java.com.example.api.controller.Class" → "controller" "main-repo.src/models/device.py::DeviceModel" → "models" """ if '::' not in fqdn: diff --git a/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py b/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py index 63678759..37a6aa1d 100644 --- a/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py +++ b/codewiki/src/be/dependency_analyzer/analysis/repo_analyzer.py @@ -141,7 +141,7 @@ def _get_namespace_from_path(self, repo_path: str) -> str: repo_path: Path to repository Returns: - Namespace string (e.g., 'openframe-frontend', 'ui-kit') + Namespace string: the repository directory's name """ return os.path.basename(os.path.abspath(repo_path).rstrip(os.sep)) diff --git a/codewiki/src/be/dependency_analyzer/ast_parser.py b/codewiki/src/be/dependency_analyzer/ast_parser.py index 88848288..513196d4 100644 --- a/codewiki/src/be/dependency_analyzer/ast_parser.py +++ b/codewiki/src/be/dependency_analyzer/ast_parser.py @@ -101,7 +101,7 @@ def _get_namespace_from_path(self, repo_path: str) -> str: repo_path: Absolute path to repository Returns: - Namespace string (e.g., 'openframe-frontend') + Namespace string: the repository directory's name """ # Use the last directory name as namespace return os.path.basename(repo_path.rstrip(os.sep)) diff --git a/codewiki/src/be/dependency_analyzer/utils/third_party.py b/codewiki/src/be/dependency_analyzer/utils/third_party.py index a495123d..2e37b02e 100644 --- a/codewiki/src/be/dependency_analyzer/utils/third_party.py +++ b/codewiki/src/be/dependency_analyzer/utils/third_party.py @@ -9,8 +9,8 @@ - minified: a `.js` / `.css` file whose average line is longer than 110 characters, GitHub Linguist's rule for minified files (a person's code averages well under that; one repository's own files averaged 21 to 78), unless it also holds at least 200 ordinary lines (non-empty, at most 110 characters): a hand-written file with one long - embedded data line (meshcentral's agents/meshcmd.js: 2,673 ordinary lines) is code, while minified libraries - have a few dozen at most (OpenLayers 33, each Bootstrap theme 11); + embedded data line has thousands of them and is code, while minified libraries have a few dozen at most + (OpenLayers 33, each Bootstrap theme 11); - a library banner: the file opens with a `/*!` comment, the form minifiers and library builds keep for the library's license (`/*! jQuery v3.7.1`, `/*! Bootstrap v5.3.3`); a project's own files open with `/**` or `//`; - a vendored folder (`vendor`, `third_party`, `bower_components`, …): matched by the walk's own path patterns. diff --git a/codewiki/src/be/prompt_template.py b/codewiki/src/be/prompt_template.py index 3e77537e..3da0d1dd 100644 --- a/codewiki/src/be/prompt_template.py +++ b/codewiki/src/be/prompt_template.py @@ -23,7 +23,7 @@ When code-graph tools are available, look such code up instead of guessing: - `resolve_code_imports`: what an import refers to and which repository defines it - `get_code_symbol`: a definition, its signature and its callers -- `find_code_symbols`: the definitions under a package, module or crate (`com.openframe.data`, `@scope/pkg`), whichever repository holds them +- `find_code_symbols`: the definitions under a package, module or crate (`com.example.data`, `@scope/pkg`), whichever repository holds them Which artifacts this repository publishes and consumes, at which versions, and who consumes it, are in ECOSYSTEM_FACTS when present: read them there rather than asking for them again. Document how THIS repository uses that code, never the other repository's internals. Tool results arrive wrapped in a `<<