mirror of
https://github.com/tiennm99/torrent2drive.git
synced 2026-08-21 00:25:16 +00:00
docs: mark the Colab approach as failed and abandoned
Colab enforces its ban on peer-to-peer file sharing by terminating the runtime once torrent traffic is detected, so the pipeline never completes a batch. The disk problem the batching design solves was never the blocker; host policy enforcement is. README, notebook intro/notes, and both docs now lead with the failure and point at a VPS or seedbox instead. Removes the Colab badge so the notebook is not one click from running. No code changes.
This commit is contained in:
@@ -1,98 +1,80 @@
|
||||
# torrent2drive
|
||||
|
||||
A single Google Colab notebook that downloads a torrent inside a Colab runtime and uploads it to
|
||||
your Google Drive **in batches**, so the torrent's total size never has to fit on the runtime's
|
||||
local disk.
|
||||
> **This does not work. Do not use it.**
|
||||
>
|
||||
> Google Colab terminates the runtime when it detects torrent traffic. Downloading torrents is
|
||||
> prohibited by Colab's [FAQ](https://research.google.com/colaboratory/faq.html), and the
|
||||
> prohibition is enforced, not just written down — the runtime is killed mid-download, so the
|
||||
> notebook never gets far enough to upload anything. The batching design below is sound on paper
|
||||
> and irrelevant in practice, because the host stops the process.
|
||||
>
|
||||
> This repository is kept as a record of an approach that was tried and failed. **Use a VPS or a
|
||||
> seedbox instead.**
|
||||
|
||||
[](https://colab.research.google.com/github/tiennm99/torrent2drive/blob/main/torrent2drive.ipynb)
|
||||
An attempt at a single Google Colab notebook that downloads a torrent inside a Colab runtime and
|
||||
uploads it to Google Drive **in batches**, so the torrent's total size never has to fit on the
|
||||
runtime's local disk.
|
||||
|
||||
## Read this before you run it
|
||||
## What actually happened
|
||||
|
||||
Colab's [FAQ](https://research.google.com/colaboratory/faq.html) prohibits **"downloading torrents
|
||||
or engaging in peer-to-peer file-sharing"** from its runtimes. That restriction is about the
|
||||
activity, not the content: it applies even to a torrent you have every right to download. Running
|
||||
this notebook on Colab breaches Colab's terms and can cost you the Google account.
|
||||
The disk problem was solvable and was solved — the sliding-window design below keeps peak local disk
|
||||
at a fixed budget regardless of torrent size. That was never the blocker.
|
||||
|
||||
If you want to do this routinely, a cheap VPS or seedbox is the honest answer — it removes the
|
||||
disk cap, the 12-hour session cap, and this restriction in one move.
|
||||
The blocker is policy enforcement. Colab prohibits **"downloading torrents or engaging in
|
||||
peer-to-peer file-sharing"** from its runtimes, and that restriction is about the activity, not the
|
||||
content: it applies even to a torrent you have every right to download. In practice the runtime gets
|
||||
killed once the download is under way, so the pipeline cannot complete a single batch reliably.
|
||||
Repeated attempts also put the Google account at risk.
|
||||
|
||||
## Quick start
|
||||
There is no fix inside the notebook. Obfuscating the traffic to evade detection would still breach
|
||||
the terms, so it is not a direction this repo will take.
|
||||
|
||||
1. Click the badge above to open `torrent2drive.ipynb` in Colab.
|
||||
2. Run cell 1 to install libtorrent and authorise Drive.
|
||||
3. Paste a magnet link into `SOURCE` in cell 3 (or upload a `.torrent` file with cell 2 and paste
|
||||
the path it prints), set `DRIVE_FOLDER` and `DISK_BUDGET_GB`, then run it.
|
||||
4. Run cell 4 to load the Drive helpers.
|
||||
5. Run cell 5. It downloads and uploads batch by batch until the torrent is in Drive.
|
||||
**The working answer is a cheap VPS or seedbox.** It removes the disk cap, the 12-hour session cap,
|
||||
and the terms problem in one move, and it can seed afterwards — which this notebook could never do.
|
||||
|
||||
## How it handles torrents bigger than the disk
|
||||
## What was built anyway
|
||||
|
||||
Every file in the torrent starts at libtorrent priority `0`, so nothing downloads. The pipeline
|
||||
then repeats until it runs out of files:
|
||||
For anyone reading the code as a reference rather than a tool:
|
||||
|
||||
1. take the next group of files whose combined size fits inside `DISK_BUDGET_GB`,
|
||||
2. raise those files to normal priority and wait for them to complete,
|
||||
3. upload each to Drive with a resumable chunked upload and verify the size Drive reports,
|
||||
4. delete them locally and drop the priorities back to `0`.
|
||||
|
||||
**Peak local disk is the budget, not the torrent size.** A 500 GB pack moves through a 20 GB window
|
||||
without ever needing 500 GB of disk.
|
||||
|
||||
The one hard limit: the **largest single file** must fit in the budget. Cell 5 aborts with a clear
|
||||
message if it does not, because a single file larger than the runtime disk needs a byte-range
|
||||
streaming approach this notebook does not implement.
|
||||
|
||||
## What it does
|
||||
|
||||
- Magnet links and `.torrent` files, via the [libtorrent](https://libtorrent.org/) Python bindings.
|
||||
- Live per-batch progress: percentage, bytes, speed, peer and seed counts, torrent state.
|
||||
- Magnet links and `.torrent` files via the [libtorrent](https://libtorrent.org/) Python bindings.
|
||||
- A per-file sliding window: every file starts at libtorrent priority `0`, and the pipeline
|
||||
repeatedly takes the next group of files fitting `DISK_BUDGET_GB`, downloads only those, uploads
|
||||
them, deletes them locally, and drops the priorities back. Peak local disk is the budget, not the
|
||||
torrent size. The one hard limit is that the largest single file must fit in the budget.
|
||||
- Resumable chunked uploads straight to the Drive REST API, with size verification before any local
|
||||
file is deleted.
|
||||
- Restart safety: re-running cell 5 indexes the destination folder and skips every file already in
|
||||
Drive at the correct size, so only the in-flight batch is ever redone.
|
||||
- A hard disk floor: the run aborts with an actionable error if free space drops below 2 GiB,
|
||||
rather than hanging on a failed write.
|
||||
file is deleted. No `drive.mount`: Colab's Drive FUSE mount stages content in
|
||||
`/root/.config/Google/DriveFS/<id>/content_cache` on the **local disk**, unbounded and without
|
||||
eviction ([colabtools#960](https://github.com/googlecolab/colabtools/issues/960)), so it would
|
||||
consume the very disk the design exists to avoid.
|
||||
- Restart safety: re-running the pipeline indexes the destination folder and skips every file
|
||||
already in Drive at the correct size.
|
||||
|
||||
## Why there is no `drive.mount`
|
||||
None of this was ever validated end to end on a live runtime, because the runtime does not survive
|
||||
long enough to validate it.
|
||||
|
||||
An earlier version wrote through Colab's Drive FUSE mount and offered a `direct-to-drive` mode for
|
||||
oversized torrents. That mode did not work: DriveFS stages content in
|
||||
`/root/.config/Google/DriveFS/<id>/content_cache` on the **local disk**, unbounded and without
|
||||
eviction ([colabtools#960](https://github.com/googlecolab/colabtools/issues/960)), so it consumed
|
||||
the very disk it existed to avoid — and DriveFS has no random-write path for the out-of-order
|
||||
pieces a torrent produces.
|
||||
|
||||
The notebook talks to the Drive REST API directly instead. Colab's own credentials authorise it, so
|
||||
there is still no API key, service account, or rclone config to set up. Full reasoning, including
|
||||
the alternatives that were considered and rejected, is in
|
||||
[docs/system-architecture.md](docs/system-architecture.md).
|
||||
|
||||
## Limits worth knowing
|
||||
## Other limits, if the terms problem did not exist
|
||||
|
||||
- **Local disk** is roughly 35–78 GB on a free runtime, not 100 GB, and a GPU runtime gets less.
|
||||
Cell 3 prints what is actually free and refuses a budget without headroom.
|
||||
- **Session length** is capped at 12 hours on the free tier, and idle runtimes are reclaimed sooner.
|
||||
Batching means a disconnect costs you one batch, not the whole download.
|
||||
- **Drive storage** is 15 GB free. A big torrent needs paid storage to land anywhere.
|
||||
- **Drive uploads** are capped at 750 GB per 24 hours on every account, personal or Workspace.
|
||||
- **No seeding.** The notebook never unchokes a peer: it deletes each file once it is safely in
|
||||
Drive and could not serve a request for it. Seeds still upload to us; other leechers will not.
|
||||
- **Drive storage** is 15 GB free; **Drive uploads** are capped at 750 GB per 24 hours on every
|
||||
account, personal or Workspace.
|
||||
- **No seeding.** The notebook deletes each file once it is in Drive and could not serve a request
|
||||
for it, so it never unchokes a peer.
|
||||
- **Magnet resolution** can take minutes on poorly seeded torrents and may never succeed from a
|
||||
datacenter IP. Cell 5 gives up after five minutes. A `.torrent` file skips the lookup.
|
||||
datacenter IP.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [docs/system-architecture.md](docs/system-architecture.md) — the constraint stack behind the
|
||||
design, cell responsibilities, failure handling, rejected alternatives, and what is still
|
||||
unverified against a live runtime.
|
||||
design, cell responsibilities, failure handling, and rejected alternatives.
|
||||
- [docs/project-changelog.md](docs/project-changelog.md) — what changed and why, including the
|
||||
breaking rewrite that removed `direct-to-drive`.
|
||||
point at which the approach was abandoned.
|
||||
|
||||
## Legal
|
||||
|
||||
Only download content you have the right to download. Beyond copyright, note the Colab restriction
|
||||
described at the top of this file. This notebook is a general purpose BitTorrent client; what you
|
||||
point it at is your responsibility.
|
||||
described at the top of this file — it is the reason this project is archived. This notebook is a
|
||||
general purpose BitTorrent client; what you point it at is your responsibility.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -1,5 +1,20 @@
|
||||
# Changelog
|
||||
|
||||
## 2026-08-02 — approach abandoned
|
||||
|
||||
The notebook does not work on Colab and will not be developed further.
|
||||
|
||||
Colab enforces its prohibition on "downloading torrents or engaging in peer-to-peer file-sharing"
|
||||
by **terminating the runtime** once torrent traffic is detected. The download is cut off before a
|
||||
batch completes, so nothing reliably reaches Drive. This is a host policy decision, not a defect in
|
||||
the pipeline: the disk problem the batching design solves was never the thing that stopped it.
|
||||
|
||||
No fix is possible within the notebook's premise. Evading detection would still breach the terms,
|
||||
so that direction is out of scope. A VPS or seedbox is the working answer.
|
||||
|
||||
Documentation only — the README and the notebook's intro and notes cells now lead with the failure;
|
||||
no code changed. The repository stands as a record of an approach that was tried and failed.
|
||||
|
||||
## Unreleased — 2026-08-01
|
||||
|
||||
Rewrite of `torrent2drive.ipynb` around a per-file sliding window. **Breaking for anyone running a
|
||||
|
||||
@@ -2,7 +2,13 @@
|
||||
|
||||
How `torrent2drive.ipynb` is put together, and why it is built this way rather than the obvious way.
|
||||
|
||||
Last reviewed: 2026-08-01
|
||||
Last reviewed: 2026-08-02
|
||||
|
||||
> **Status: abandoned.** The design below is a record, not a recommendation. Colab enforces its
|
||||
> ban on peer-to-peer file sharing by terminating the runtime once torrent traffic is detected, so
|
||||
> the pipeline never completes a run regardless of how well it bounds disk usage. The binding
|
||||
> constraint turned out to be host policy enforcement, not the constraint stack this design was
|
||||
> built around. See [project-changelog.md](project-changelog.md).
|
||||
|
||||
## The problem
|
||||
|
||||
@@ -22,7 +28,7 @@ Every design decision below follows from one of these. All were verified on 2026
|
||||
| Drive storage | 15 GB free | Large torrents require paid storage; not something the notebook can fix. |
|
||||
| Drive upload cap | 750 GB / 24 h, all account types | A very large torrent may need more than one day. Failures must not lose local progress silently. |
|
||||
| Inbound connectivity | None on a Colab runtime | UPnP and NAT-PMP are no-ops. Only outbound peers and DHT work. |
|
||||
| Colab terms | P2P file sharing prohibited outright, regardless of content ([FAQ](https://research.google.com/colaboratory/faq.html)) | Documented prominently. Not a technical constraint, but it is the reason a VPS is the better long-term host. |
|
||||
| Colab terms | P2P file sharing prohibited outright, regardless of content ([FAQ](https://research.google.com/colaboratory/faq.html)), and **enforced by killing the runtime** | Fatal. Observed 2026-08-02: the runtime is terminated mid-download, so no batch completes. This is what ended the project; a VPS or seedbox is the only host where the rest of the design can run. |
|
||||
|
||||
## Core design: per-file sliding window
|
||||
|
||||
@@ -111,7 +117,7 @@ key, service account, or rclone config.
|
||||
| `rclone mount --vfs-cache-mode writes` | Tempting trap. rclone cannot evict files that are still open, and libtorrent holds every file open until the torrent completes, so the cache grows to the full torrent size and `--vfs-cache-max-size` stalls instead of bounding it. |
|
||||
| `rclone copy` for the upload leg | Works, and is strictly better than copying over FUSE, but it needs an OAuth remote config. The Drive API gives the same chunking and retries using credentials Colab already has. |
|
||||
| Sequential download + `fallocate(FALLOC_FL_PUNCH_HOLE)` | The only approach that handles a single file larger than the disk: download sequentially into a sparse file, upload the completed prefix, punch holes behind the upload pointer. Costs peer efficiency, forbids re-check and seeding, and depends on punch-hole support on Colab's `/content` filesystem, which is unverified. Worth revisiting only if single oversized files become a real need. |
|
||||
| A VPS or seedbox | Genuinely the better answer for routine use — it removes the disk cap, the session cap, and the terms problem at once. Out of scope for a repo whose premise is a Colab notebook. |
|
||||
| A VPS or seedbox | Rejected at the time as out of scope for a repo whose premise is a Colab notebook. **This was the wrong call** — it removes the disk cap, the session cap, and the terms problem at once, and it is now the recommendation. |
|
||||
|
||||
## Verification
|
||||
|
||||
@@ -120,13 +126,12 @@ off-by-one, oversized single file, the peak-disk invariant), pad filtering with
|
||||
byte formatting, and Drive query escaping. The notebook itself is checked with
|
||||
`nbformat.validate()` and `ast.parse` over every code cell.
|
||||
|
||||
**Unverified against a live runtime.** The libtorrent and Drive integration has not been executed on
|
||||
Colab. Specifically unproven:
|
||||
**Never verified against a live runtime, and now unverifiable on Colab.** Attempting a real run is
|
||||
what surfaced the enforcement: the runtime is killed during the download, before the integration
|
||||
gets far enough to test. These remain unproven and will stay that way unless the notebook is ported
|
||||
to a host that permits the traffic:
|
||||
|
||||
- `handle.clear_error()` availability in the installed binding version (guarded with `getattr`).
|
||||
- `unchoke_slots_limit: 0` genuinely suppressing all uploads.
|
||||
- Whether `auth.authenticate_user()` credentials carry sufficient Drive scope for `files().create`
|
||||
in the current Colab image.
|
||||
|
||||
First real run should be a small multi-file torrent with `DISK_BUDGET_GB` set below its total size,
|
||||
so at least two batches are exercised.
|
||||
|
||||
+21
-10
@@ -22,23 +22,29 @@
|
||||
"id": "intro"
|
||||
},
|
||||
"source": [
|
||||
"# torrent2drive\n",
|
||||
"# torrent2drive — does not work\n",
|
||||
"\n",
|
||||
"Download a torrent inside a Google Colab runtime and upload it to your Google Drive **in batches**,\n",
|
||||
"so the torrent's total size never has to fit on the runtime's local disk.\n",
|
||||
"> **This approach failed. Do not run it.**\n",
|
||||
">\n",
|
||||
"> Google Colab prohibits \"downloading torrents or engaging in peer-to-peer file-sharing\" from its\n",
|
||||
"> runtimes ([FAQ](https://research.google.com/colaboratory/faq.html)), and enforces it: the runtime\n",
|
||||
"> is killed once torrent traffic is detected, so the pipeline never finishes. Repeated attempts can\n",
|
||||
"> also cost you the Google account.\n",
|
||||
">\n",
|
||||
"> The notebook is kept as a record of an approach that was tried and failed. Use a VPS or a seedbox\n",
|
||||
"> instead — it removes the disk cap, the session cap, and the terms problem at once.\n",
|
||||
"\n",
|
||||
"**Run the cells top to bottom.**\n",
|
||||
"What it was meant to do: download a torrent inside a Colab runtime and upload it to Google Drive\n",
|
||||
"**in batches**, so the torrent's total size never has to fit on the runtime's local disk. The\n",
|
||||
"batching design holds up on paper; the host stops the process before that matters.\n",
|
||||
"\n",
|
||||
"The cells, for reading rather than running:\n",
|
||||
"\n",
|
||||
"1. Install libtorrent and authorise Drive\n",
|
||||
"2. (Optional) upload a `.torrent` file\n",
|
||||
"3. Fill in the configuration form\n",
|
||||
"4. Load the Drive helpers\n",
|
||||
"5. Download and upload\n",
|
||||
"\n",
|
||||
"> **Read this first.** Colab's [FAQ](https://research.google.com/colaboratory/faq.html) prohibits\n",
|
||||
"> \"downloading torrents or engaging in peer-to-peer file-sharing\" from its runtimes — regardless of\n",
|
||||
"> whether the content itself is legal. Running this notebook on Colab breaches those terms and can\n",
|
||||
"> cost you the Google account. Only download content you have the right to download."
|
||||
"5. Download and upload"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -497,6 +503,11 @@
|
||||
"id": "notes"
|
||||
},
|
||||
"source": [
|
||||
"## Status: abandoned\n",
|
||||
"\n",
|
||||
"Everything below describes the intended behaviour. None of it survives contact with Colab, which\n",
|
||||
"kills the runtime when it detects torrent traffic. Read it as a design record, not as instructions.\n",
|
||||
"\n",
|
||||
"## How the batching works\n",
|
||||
"\n",
|
||||
"Every file in the torrent starts at priority `0`, so libtorrent downloads nothing. The pipeline\n",
|
||||
|
||||
Reference in New Issue
Block a user