Rendering markdown & diagrams...
1 # NEURALOS SPECIFICATION: MAGNET VAULT, STREMIO STREAMING & AI VIDEO GENERATION
2 **DOCUMENT ID:** SPEC-005
3 **CLASSIFICATION:** SYSTEM ARCHITECTURE & IMPLEMENTATION SPEC
4 **COMMAND AUTHORITY:** CHIEF ARCHITECT (SHUBHAM)
5 **STATUS:** PENDING ARCHITECTURAL APPROVAL
6
7 ---
8
9 ## 1. Executive Summary & Core Philosophy
10
11 This architecture shifts our media and intelligence strategy to a **zero-waste, on-demand model**:
12
13 1. **Magnet Vault Over Tor (No Wasted Disk Space)**:
14 - We do **NOT** blindly download large multi-gigabyte video files upfront.
15 - The agentic crawler queries multiple torrent indexers (YTS, PirateBay/APBay, 1337x) over **Tor SOCKS5h (`127.0.0.1:9050`)** to bypass ISP censorship and extract metadata + magnet links into a lightweight SQLite vault (`magnets.db`).
16 2. **Stremio-Compatible Sequential Streaming**:
17 - Instead of waiting for full downloads, we stream video chunks sequentially over HTTP directly into HTML5 players, VLC, or native Stremio apps.
18 - We expose a lightweight private **Stremio Addon (`/manifest.json`)** on Dell. You can add this URL to Stremio on your Mac, iPhone, or Smart TV, and your agent's curated movies appear directly inside Stremio with full metadata, subtitles, and instant streaming!
19 3. **Save-to-Disk on Demand (ROE-01 Strict)**:
20 - When you explicitly click *"Save / Download to Disk"*, the magnet is sent to `transmission-daemon` on Dell to permanently save into `/mnt/hdd1/MOvies/` (1TB HDD).
21 - Shubham's personal Drive D (`/mnt/drive_d`) remains **100% read-only**.
22 4. **Dual-Tier AI Video Generation**:
23 - **Dell (Node-01)**: CPU-only, legacy 2GB GPU → Serves as the media coordinator, storage anchor, and `ffmpeg` programmatic editor.
24 - **Mac (16GB Apple Silicon)**: Metal/MLX-accelerated local generation using compact open-weight models (**`LTX-Video` 2B** or **`Wan2.1-1.3B`**).
25 - **Cloud API Gateway**: Integrated into PicoClaw for studio-grade cinematic video generation (Runway, Luma, Fal.ai, Wan 14B) when high resolution is needed.
26
27 ---
28
29 ## 2. Architecture Diagram
30
31 ```mermaid
32 flowchart TD
33 subgraph DellNode01["Dell Node-01 (Tactical Body & Storage Anchor)"]
34 subgraph TorSubsystem["Anonymity Layer"]
35 TorDaemon["Tor Daemon (127.0.0.1:9050)\nSOCKS5h Remote DNS"]
36 end
37
38 subgraph SearchEngine["Search & Indexing Engine"]
39 MagnetVault["Magnet Vault (/var/neural_memory/data/magnets.db)\n• YTS Official JSON API\n• APBay / PirateBay Index\n• 1337x Scraper"]
40 end
41
42 subgraph StreamingEngine["Streaming & Addon Subsystem"]
43 StremioAddon["Private Stremio Addon (:7000/manifest.json)\n• Catalogs agent-found media\n• Exposes streams to Stremio clients"]
44 TorrentStream["Sequential Torrent Streamer\n• Streams video chunks over HTTP\n• Zero disk bloat"]
45 TransDaemon["transmission-daemon (:9091)\n• Saves selected media permanently"]
46 end
47
48 subgraph StorageSubsystem["Storage Subsystem"]
49 HDD1["/mnt/hdd1/MOvies/\n/mnt/hdd1/downloads/\n(1TB Mass Storage Drive)"]
50 DriveD["/mnt/drive_d (sda5)\n(ROE-01: READ-ONLY PROTECTED)"]
51 end
52
53 subgraph Agent["PicoClaw Agent Gateway (:18790)"]
54 PicoClaw["PicoClaw Daemon"]
55 PicoSkills["Skills:\n• torrent-search\n• github-crawl\n• video-gen\n• tor-browse"]
56 end
57 end
58
59 subgraph MasterMac["Apple Silicon Mac (Master Brain)"]
60 OllamaLLM["DeepSeek-R1-Distill-Qwen-7B\n(Metal GPU Accelerated)"]
61 LocalVideoGen["Local Video Gen (MLX)\n• LTX-Video 2B / Wan2.1 1.3B\n(Quantized on Metal)"]
62 StremioApp["Stremio App / Web UI\n(Mac, iPhone, Smart TV)"]
63 WebDashboard["Media Vault Dashboard\nhttp://dell-ss.lan:8080\nhttp://100.93.213.110:8080"]
64 end
65
66 PicoClaw --> MagnetVault
67 MagnetVault --> TorDaemon
68 TorDaemon --> Internet["Tor Network"]
69 PicoClaw --> TorrentStream
70 TorrentStream --> StremioAddon
71 StremioAddon --> StremioApp
72 StremioAddon --> WebDashboard
73 TransDaemon --> HDD1
74 WebDashboard --> DriveD
75 WebDashboard --> HDD1
76 PicoClaw -.-> OllamaLLM
77 PicoClaw -.-> LocalVideoGen
78 ```
79
80 ---
81
82 ## 3. Deep Dive: How Stremio Works & Why It Fits Perfectly
83
84 ### What is Stremio?
85 Stremio is an open-source media aggregation platform that completely decouples **metadata catalogs**, **streaming sources**, and the **video playback client**.
86
87 ### How Stremio Streams Torrents:
88 1. **Sequential Torrent Engine**: Standard torrent clients download the "rarest piece first". Stremio's streaming engine downloads pieces **in sequential order** (head of file → tail of file). As soon as the first 1%–2% buffer is ready, video playback begins immediately.
89 2. **The Open Addon Protocol**:
90 - An Addon is simply an HTTP server that returns a `manifest.json`.
91 - **`GET /manifest.json`**: Tells Stremio what types of media (movies, series) and search features it supports.
92 - **`GET /catalog/movie/agent_picks.json`**: Lists movie titles, IMDb ratings, and posters curated by your Dell agent.
93 - **`GET /stream/movie/{id}.json`**: Returns the magnet link or streaming URL.
94 3. **Multi-Device Sync**:
95 - By creating a lightweight Stremio Addon on Dell (`http://100.93.213.110:7000/manifest.json`), you can paste that link once into Stremio on your Mac, iPhone, or TV.
96 - Your agent's findings automatically appear in Stremio's native theater UI with hardware acceleration, automated subtitle downloads (OpenSubtitles), and audio track switching!
97
98 ---
99
100 ## 4. Deep Dive: AI Video Generation Capabilities
101
102 ### Hardware Reality Check:
103 * **Dell Node-01**: 3rd Gen Intel Core CPU, 8GB DDR3 RAM, 2GB legacy AMD GPU. Dell **cannot** generate diffusion videos locally (would take hours per frame).
104 * **Mac (16GB Apple Silicon)**: Unified memory and 8-core/16-core GPU with Apple's **MLX** framework **can** run lightweight distilled video models!
105
106 ### Recommended Video Generation Tiering:
107
108 | Tier | Engine / Model | Hardware | Video Specs | Best For |
109 | :--- | :--- | :--- | :--- | :--- |
110 | **Tier 1: Local Real-Time (Free)** | **`LTX-Video` (2B)** via **MLX** (`mlx-video`) | Mac M-Series (16GB RAM) | 720p @ 24fps, 5–10s clips | Rapid prototyping, motion testing, memes, local offline generation. |
111 | **Tier 2: High-Quality Open-Weights** | **`Wan2.1-T2V-1.3B`** | Mac M-Series (quantized) | 480p/720p cinematic motion | Narrative clips with complex prompt understanding. |
112 | **Tier 3: Cloud Studio-Grade** | **Fal.ai / Replicate / Runway Gen-3** | Cloud API (routed via PicoClaw) | 1080p/4K @ 30fps, photorealistic | Production-quality video clips when prompt demands photorealism. |
113 | **Tier 4: Programmatic Video Engine** | **`ffmpeg` + Python** | Dell Node-01 | 1080p 60fps unlimited | Stitches generated clips, adds audio tracks, applies Ken-Burns pan/zoom, burns subtitles. |
114
115 ---
116
117 ## 5. Required Feature Set & Implementation Scope
118
119 ### Feature 1: Magnet Vault & Tor Searcher (`magnet_vault.py`)
120 - Background search over Tor SOCKS5h (`127.0.0.1:9050`).
121 - Scrapes:
122 - **YTS Official REST API**: Instant JSON for 720p/1080p/4K releases with seeder counts.
123 - **APBay / PirateBay Index**: Direct JSON API for general media, TV series, and software.
124 - **1337x**: Fallback parser for high-seed releases.
125 - Stores results in `/var/neural_memory/data/magnets.db` with columns: `title`, `year`, `quality`, `size_bytes`, `seeders`, `magnet_uri`, `source`, `added_at`.
126
127 ### Feature 2: Stremio Addon on Dell (`stremio_addon.py`)
128 - Lightweight Python HTTP server on port `7000`.
129 - Implements the official Stremio Addon Protocol v3:
130 - `manifest.json`: Defines resource `stream` and `catalog`.
131 - Serves magnet links directly to any Stremio player over Tailscale (`http://100.93.213.110:7000`).
132
133 ### Feature 3: Web Dashboard Integration (`http://dell-ss.lan:8080`)
134 - Adds a **"Magnet Vault & Tor Search"** tab directly to our existing dashboard.
135 - Features:
136 - Search bar to query movies/torrents over Tor.
137 - Display search results with title, size, seeder badge, and two action buttons:
138 - `▶ Stream Now` → Opens instant video player.
139 - `⬇ Save to 1TB HDD` → Sends magnet to `transmission-daemon` to persist in `/mnt/hdd1/MOvies/`.
140
141 ### Feature 4: PicoClaw Agent Skills
142 1. **`torrent-search`**: Agent skill to find media, compare seeder health, and store magnet links.
143 2. **`github-crawl`**: Agent skill to search repos, extract READMEs, and analyze architectures.
144 3. **`video-gen`**: Agent skill that takes a prompt and triggers local MLX generation (on Mac) or Cloud API, saving the final `.mp4` into `/mnt/hdd1/MOvies/`.
145
146 ---
147
148 ## 6. Stremio Ecosystem Addon Audit & Federation Strategy
149
150 ### 6.1 Audit of Investigated Manifests
151
152 | Manifest URL | Addon Identity | Status & Health | Core Utility in NeuralOS Architecture |
153 | :--- | :--- | :--- | :--- |
154 | `https://torrentio.strem.fun/manifest.json` | **Torrentio** | **CRITICAL (Online)** | Multi-provider scraper (YTS, 1337x, TPB, RuTracker, Galaxy). Ingested over Tor into `magnet_vault.py` for instant multi-quality stream resolution. |
155 | `https://v3-cinemeta.strem.io/manifest.json` | **Cinemeta** | **CRITICAL (Online)** | Official Stremio metadata engine. Used by our crawler as the primary fuzzy title → IMDb ID & poster artwork resolver (`/catalog/movie/top/search=...json`). |
156 | `https://torrentsdb.com/manifest.json` | **TorrentsDB** | **ACTIVE (Online)** | Secondary DHT & Knaben torrent indexer. Used as an automatic fallback when Torrentio experiences rate limiting. |
157 | `https://caching.stremio.net/publicdomainmovies.now.sh/manifest.json` | **Public Domain** | **ACTIVE (Online)** | 500+ legal, public domain classic cinema titles. Useful as an offline benchmark and copyright-free test harness. |
158 | `https://thepiratebay-plus.strem.fun/manifest.json` | **TPB Plus** | **DEPRECATED (502)** | Stale/dead community deployment. Do not use. Replaced by our native Tor-routed PirateBay parser in `magnet_vault.py`. |
159 | `https://torrent-catalogs.strem.fun/manifest.json` | **Catalogs** | **UNRELIABLE (Timeout)** | Stale community scraper. Replaced by our Cinemeta + YTS catalog pipeline. |
160
161 ### 6.2 The Federation Flow
162
163 ```mermaid
164 flowchart LR
165 UserQuery["User Search: 'Oppenheimer'"] --> Cinemeta["Cinemeta API\n(v3-cinemeta.strem.io)"]
166 Cinemeta --> ExactMeta["Exact Metadata:\n• tt15398776\n• 2023 | Christopher Nolan\n• Metahub Poster"]
167 ExactMeta --> TorSOCKS["Tor SOCKS5h\n(127.0.0.1:9050)"]
168 TorSOCKS --> Torrentio["Torrentio Stream Aggregator\n(torrentio.strem.fun)"]
169 TorSOCKS --> TorrentsDB["TorrentsDB Fallback\n(torrentsdb.com)"]
170 TorSOCKS --> YTS["YTS Direct REST API\n(movies-api.accel.li)"]
171 Torrentio --> VaultDB["SQLite Magnet Vault\n(/var/neural_memory/data/magnets.db)"]
172 TorrentsDB --> VaultDB
173 YTS --> VaultDB
174 VaultDB --> StremioAddon["NeuralOS Stremio Addon (:7000)\n(/manifest.json, /catalog, /stream)"]
175 VaultDB --> Dashboard["Web Dashboard (:8080)\n(/read, /media, /stream)"]
176 ```
177