diff --git a/README.md b/README.md index 80e2c92..ef2bfd6 100644 --- a/README.md +++ b/README.md @@ -1,98 +1,80 @@ # torrent2drive -A single Google Colab notebook that downloads a torrent inside a Colab runtime and uploads it to -your Google Drive **in batches**, so the torrent's total size never has to fit on the runtime's -local disk. +> **This does not work. Do not use it.** +> +> Google Colab terminates the runtime when it detects torrent traffic. Downloading torrents is +> prohibited by Colab's [FAQ](https://research.google.com/colaboratory/faq.html), and the +> prohibition is enforced, not just written down — the runtime is killed mid-download, so the +> notebook never gets far enough to upload anything. The batching design below is sound on paper +> and irrelevant in practice, because the host stops the process. +> +> This repository is kept as a record of an approach that was tried and failed. **Use a VPS or a +> seedbox instead.** -[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tiennm99/torrent2drive/blob/main/torrent2drive.ipynb) +An attempt at a single Google Colab notebook that downloads a torrent inside a Colab runtime and +uploads it to Google Drive **in batches**, so the torrent's total size never has to fit on the +runtime's local disk. -## Read this before you run it +## What actually happened -Colab's [FAQ](https://research.google.com/colaboratory/faq.html) prohibits **"downloading torrents -or engaging in peer-to-peer file-sharing"** from its runtimes. That restriction is about the -activity, not the content: it applies even to a torrent you have every right to download. Running -this notebook on Colab breaches Colab's terms and can cost you the Google account. +The disk problem was solvable and was solved — the sliding-window design below keeps peak local disk +at a fixed budget regardless of torrent size. That was never the blocker. -If you want to do this routinely, a cheap VPS or seedbox is the honest answer — it removes the -disk cap, the 12-hour session cap, and this restriction in one move. +The blocker is policy enforcement. Colab prohibits **"downloading torrents or engaging in +peer-to-peer file-sharing"** from its runtimes, and that restriction is about the activity, not the +content: it applies even to a torrent you have every right to download. In practice the runtime gets +killed once the download is under way, so the pipeline cannot complete a single batch reliably. +Repeated attempts also put the Google account at risk. -## Quick start +There is no fix inside the notebook. Obfuscating the traffic to evade detection would still breach +the terms, so it is not a direction this repo will take. -1. Click the badge above to open `torrent2drive.ipynb` in Colab. -2. Run cell 1 to install libtorrent and authorise Drive. -3. Paste a magnet link into `SOURCE` in cell 3 (or upload a `.torrent` file with cell 2 and paste - the path it prints), set `DRIVE_FOLDER` and `DISK_BUDGET_GB`, then run it. -4. Run cell 4 to load the Drive helpers. -5. Run cell 5. It downloads and uploads batch by batch until the torrent is in Drive. +**The working answer is a cheap VPS or seedbox.** It removes the disk cap, the 12-hour session cap, +and the terms problem in one move, and it can seed afterwards — which this notebook could never do. -## How it handles torrents bigger than the disk +## What was built anyway -Every file in the torrent starts at libtorrent priority `0`, so nothing downloads. The pipeline -then repeats until it runs out of files: +For anyone reading the code as a reference rather than a tool: -1. take the next group of files whose combined size fits inside `DISK_BUDGET_GB`, -2. raise those files to normal priority and wait for them to complete, -3. upload each to Drive with a resumable chunked upload and verify the size Drive reports, -4. delete them locally and drop the priorities back to `0`. - -**Peak local disk is the budget, not the torrent size.** A 500 GB pack moves through a 20 GB window -without ever needing 500 GB of disk. - -The one hard limit: the **largest single file** must fit in the budget. Cell 5 aborts with a clear -message if it does not, because a single file larger than the runtime disk needs a byte-range -streaming approach this notebook does not implement. - -## What it does - -- Magnet links and `.torrent` files, via the [libtorrent](https://libtorrent.org/) Python bindings. -- Live per-batch progress: percentage, bytes, speed, peer and seed counts, torrent state. +- Magnet links and `.torrent` files via the [libtorrent](https://libtorrent.org/) Python bindings. +- A per-file sliding window: every file starts at libtorrent priority `0`, and the pipeline + repeatedly takes the next group of files fitting `DISK_BUDGET_GB`, downloads only those, uploads + them, deletes them locally, and drops the priorities back. Peak local disk is the budget, not the + torrent size. The one hard limit is that the largest single file must fit in the budget. - Resumable chunked uploads straight to the Drive REST API, with size verification before any local - file is deleted. -- Restart safety: re-running cell 5 indexes the destination folder and skips every file already in - Drive at the correct size, so only the in-flight batch is ever redone. -- A hard disk floor: the run aborts with an actionable error if free space drops below 2 GiB, - rather than hanging on a failed write. + file is deleted. No `drive.mount`: Colab's Drive FUSE mount stages content in + `/root/.config/Google/DriveFS//content_cache` on the **local disk**, unbounded and without + eviction ([colabtools#960](https://github.com/googlecolab/colabtools/issues/960)), so it would + consume the very disk the design exists to avoid. +- Restart safety: re-running the pipeline indexes the destination folder and skips every file + already in Drive at the correct size. -## Why there is no `drive.mount` +None of this was ever validated end to end on a live runtime, because the runtime does not survive +long enough to validate it. -An earlier version wrote through Colab's Drive FUSE mount and offered a `direct-to-drive` mode for -oversized torrents. That mode did not work: DriveFS stages content in -`/root/.config/Google/DriveFS//content_cache` on the **local disk**, unbounded and without -eviction ([colabtools#960](https://github.com/googlecolab/colabtools/issues/960)), so it consumed -the very disk it existed to avoid — and DriveFS has no random-write path for the out-of-order -pieces a torrent produces. - -The notebook talks to the Drive REST API directly instead. Colab's own credentials authorise it, so -there is still no API key, service account, or rclone config to set up. Full reasoning, including -the alternatives that were considered and rejected, is in -[docs/system-architecture.md](docs/system-architecture.md). - -## Limits worth knowing +## Other limits, if the terms problem did not exist - **Local disk** is roughly 35–78 GB on a free runtime, not 100 GB, and a GPU runtime gets less. - Cell 3 prints what is actually free and refuses a budget without headroom. - **Session length** is capped at 12 hours on the free tier, and idle runtimes are reclaimed sooner. - Batching means a disconnect costs you one batch, not the whole download. -- **Drive storage** is 15 GB free. A big torrent needs paid storage to land anywhere. -- **Drive uploads** are capped at 750 GB per 24 hours on every account, personal or Workspace. -- **No seeding.** The notebook never unchokes a peer: it deletes each file once it is safely in - Drive and could not serve a request for it. Seeds still upload to us; other leechers will not. +- **Drive storage** is 15 GB free; **Drive uploads** are capped at 750 GB per 24 hours on every + account, personal or Workspace. +- **No seeding.** The notebook deletes each file once it is in Drive and could not serve a request + for it, so it never unchokes a peer. - **Magnet resolution** can take minutes on poorly seeded torrents and may never succeed from a - datacenter IP. Cell 5 gives up after five minutes. A `.torrent` file skips the lookup. + datacenter IP. ## Documentation - [docs/system-architecture.md](docs/system-architecture.md) — the constraint stack behind the - design, cell responsibilities, failure handling, rejected alternatives, and what is still - unverified against a live runtime. + design, cell responsibilities, failure handling, and rejected alternatives. - [docs/project-changelog.md](docs/project-changelog.md) — what changed and why, including the - breaking rewrite that removed `direct-to-drive`. + point at which the approach was abandoned. ## Legal Only download content you have the right to download. Beyond copyright, note the Colab restriction -described at the top of this file. This notebook is a general purpose BitTorrent client; what you -point it at is your responsibility. +described at the top of this file — it is the reason this project is archived. This notebook is a +general purpose BitTorrent client; what you point it at is your responsibility. ## License diff --git a/docs/project-changelog.md b/docs/project-changelog.md index 39ed84e..a48a537 100644 --- a/docs/project-changelog.md +++ b/docs/project-changelog.md @@ -1,5 +1,20 @@ # Changelog +## 2026-08-02 — approach abandoned + +The notebook does not work on Colab and will not be developed further. + +Colab enforces its prohibition on "downloading torrents or engaging in peer-to-peer file-sharing" +by **terminating the runtime** once torrent traffic is detected. The download is cut off before a +batch completes, so nothing reliably reaches Drive. This is a host policy decision, not a defect in +the pipeline: the disk problem the batching design solves was never the thing that stopped it. + +No fix is possible within the notebook's premise. Evading detection would still breach the terms, +so that direction is out of scope. A VPS or seedbox is the working answer. + +Documentation only — the README and the notebook's intro and notes cells now lead with the failure; +no code changed. The repository stands as a record of an approach that was tried and failed. + ## Unreleased — 2026-08-01 Rewrite of `torrent2drive.ipynb` around a per-file sliding window. **Breaking for anyone running a diff --git a/docs/system-architecture.md b/docs/system-architecture.md index 8fded63..99799f7 100644 --- a/docs/system-architecture.md +++ b/docs/system-architecture.md @@ -2,7 +2,13 @@ How `torrent2drive.ipynb` is put together, and why it is built this way rather than the obvious way. -Last reviewed: 2026-08-01 +Last reviewed: 2026-08-02 + +> **Status: abandoned.** The design below is a record, not a recommendation. Colab enforces its +> ban on peer-to-peer file sharing by terminating the runtime once torrent traffic is detected, so +> the pipeline never completes a run regardless of how well it bounds disk usage. The binding +> constraint turned out to be host policy enforcement, not the constraint stack this design was +> built around. See [project-changelog.md](project-changelog.md). ## The problem @@ -22,7 +28,7 @@ Every design decision below follows from one of these. All were verified on 2026 | Drive storage | 15 GB free | Large torrents require paid storage; not something the notebook can fix. | | Drive upload cap | 750 GB / 24 h, all account types | A very large torrent may need more than one day. Failures must not lose local progress silently. | | Inbound connectivity | None on a Colab runtime | UPnP and NAT-PMP are no-ops. Only outbound peers and DHT work. | -| Colab terms | P2P file sharing prohibited outright, regardless of content ([FAQ](https://research.google.com/colaboratory/faq.html)) | Documented prominently. Not a technical constraint, but it is the reason a VPS is the better long-term host. | +| Colab terms | P2P file sharing prohibited outright, regardless of content ([FAQ](https://research.google.com/colaboratory/faq.html)), and **enforced by killing the runtime** | Fatal. Observed 2026-08-02: the runtime is terminated mid-download, so no batch completes. This is what ended the project; a VPS or seedbox is the only host where the rest of the design can run. | ## Core design: per-file sliding window @@ -111,7 +117,7 @@ key, service account, or rclone config. | `rclone mount --vfs-cache-mode writes` | Tempting trap. rclone cannot evict files that are still open, and libtorrent holds every file open until the torrent completes, so the cache grows to the full torrent size and `--vfs-cache-max-size` stalls instead of bounding it. | | `rclone copy` for the upload leg | Works, and is strictly better than copying over FUSE, but it needs an OAuth remote config. The Drive API gives the same chunking and retries using credentials Colab already has. | | Sequential download + `fallocate(FALLOC_FL_PUNCH_HOLE)` | The only approach that handles a single file larger than the disk: download sequentially into a sparse file, upload the completed prefix, punch holes behind the upload pointer. Costs peer efficiency, forbids re-check and seeding, and depends on punch-hole support on Colab's `/content` filesystem, which is unverified. Worth revisiting only if single oversized files become a real need. | -| A VPS or seedbox | Genuinely the better answer for routine use — it removes the disk cap, the session cap, and the terms problem at once. Out of scope for a repo whose premise is a Colab notebook. | +| A VPS or seedbox | Rejected at the time as out of scope for a repo whose premise is a Colab notebook. **This was the wrong call** — it removes the disk cap, the session cap, and the terms problem at once, and it is now the recommendation. | ## Verification @@ -120,13 +126,12 @@ off-by-one, oversized single file, the peak-disk invariant), pad filtering with byte formatting, and Drive query escaping. The notebook itself is checked with `nbformat.validate()` and `ast.parse` over every code cell. -**Unverified against a live runtime.** The libtorrent and Drive integration has not been executed on -Colab. Specifically unproven: +**Never verified against a live runtime, and now unverifiable on Colab.** Attempting a real run is +what surfaced the enforcement: the runtime is killed during the download, before the integration +gets far enough to test. These remain unproven and will stay that way unless the notebook is ported +to a host that permits the traffic: - `handle.clear_error()` availability in the installed binding version (guarded with `getattr`). - `unchoke_slots_limit: 0` genuinely suppressing all uploads. - Whether `auth.authenticate_user()` credentials carry sufficient Drive scope for `files().create` in the current Colab image. - -First real run should be a small multi-file torrent with `DISK_BUDGET_GB` set below its total size, -so at least two batches are exercised. diff --git a/torrent2drive.ipynb b/torrent2drive.ipynb index 844f468..27dc507 100644 --- a/torrent2drive.ipynb +++ b/torrent2drive.ipynb @@ -22,23 +22,29 @@ "id": "intro" }, "source": [ - "# torrent2drive\n", + "# torrent2drive — does not work\n", "\n", - "Download a torrent inside a Google Colab runtime and upload it to your Google Drive **in batches**,\n", - "so the torrent's total size never has to fit on the runtime's local disk.\n", + "> **This approach failed. Do not run it.**\n", + ">\n", + "> Google Colab prohibits \"downloading torrents or engaging in peer-to-peer file-sharing\" from its\n", + "> runtimes ([FAQ](https://research.google.com/colaboratory/faq.html)), and enforces it: the runtime\n", + "> is killed once torrent traffic is detected, so the pipeline never finishes. Repeated attempts can\n", + "> also cost you the Google account.\n", + ">\n", + "> The notebook is kept as a record of an approach that was tried and failed. Use a VPS or a seedbox\n", + "> instead — it removes the disk cap, the session cap, and the terms problem at once.\n", "\n", - "**Run the cells top to bottom.**\n", + "What it was meant to do: download a torrent inside a Colab runtime and upload it to Google Drive\n", + "**in batches**, so the torrent's total size never has to fit on the runtime's local disk. The\n", + "batching design holds up on paper; the host stops the process before that matters.\n", + "\n", + "The cells, for reading rather than running:\n", "\n", "1. Install libtorrent and authorise Drive\n", "2. (Optional) upload a `.torrent` file\n", "3. Fill in the configuration form\n", "4. Load the Drive helpers\n", - "5. Download and upload\n", - "\n", - "> **Read this first.** Colab's [FAQ](https://research.google.com/colaboratory/faq.html) prohibits\n", - "> \"downloading torrents or engaging in peer-to-peer file-sharing\" from its runtimes — regardless of\n", - "> whether the content itself is legal. Running this notebook on Colab breaches those terms and can\n", - "> cost you the Google account. Only download content you have the right to download." + "5. Download and upload" ] }, { @@ -497,6 +503,11 @@ "id": "notes" }, "source": [ + "## Status: abandoned\n", + "\n", + "Everything below describes the intended behaviour. None of it survives contact with Colab, which\n", + "kills the runtime when it detects torrent traffic. Read it as a design record, not as instructions.\n", + "\n", "## How the batching works\n", "\n", "Every file in the torrent starts at priority `0`, so libtorrent downloads nothing. The pipeline\n",