Full-featured BitTorrent client package and utilities
Go to file
Matt Joiner 5501f994ca
Move a bunch of stuff into subpackages
The core package is very large now, and often only parts of it are needed.
2022-11-15 23:31:26 +11:00
.circleci Don't bench with -race 2021-02-24 13:48:05 +11:00
.github Fix webrtc logging for JS build 2022-04-11 14:53:02 +10:00
analysis Add peer upload order analysis 2021-01-28 16:31:13 +11:00
bencode Add bencode.Bytes.GoString 2022-03-09 20:57:23 +11:00
cmd Retrive file via HTTP URL in `metainfo.go` (#778) 2022-09-26 10:01:51 +10:00
common Got file storage working with segment index 2020-06-01 00:14:51 +10:00
dialer Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
fs fs/test.sh: Fail if hash does not match 2022-02-10 09:20:23 +11:00
internal internal/testutil.Info: Support multi-file Torrents 2022-06-14 16:01:47 +10:00
iplist gofumpt 2021-11-08 14:47:01 +11:00
logonce Add a prefix to logonce logger output 2018-09-03 11:48:31 +10:00
metainfo Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
mmap_span Do torrent storage flush on piece completion (#755) 2022-07-07 15:46:27 +10:00
mse gofumpt -extra 2021-12-23 14:00:00 +11:00
peer_protocol gofumpt -extra 2021-12-23 14:00:00 +11:00
request-strategy Update tidwall/btree (0.7.2 -> 1.3.1) (#757) 2022-06-22 14:16:20 +10:00
segments gofumpt 2021-11-08 14:47:01 +11:00
smartban Implement smart ban using generics 2022-01-22 18:43:04 +11:00
storage Do torrent storage flush on piece completion (#755) 2022-07-07 15:46:27 +10:00
test Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
testdata Add sintel.torrent 2021-12-12 15:01:50 +11:00
tracker Add ability to set DialContext/ListenPacket for tracker announcements (#760) 2022-07-07 15:51:58 +10:00
typed-roaring Use reusable roaring iterators 2022-05-12 10:42:16 +10:00
types Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
util/dirwatch test: use `T.TempDir` to create temporary test directory (#718) 2022-02-02 18:26:46 +11:00
version Update and expose default client identifiers 2021-06-25 11:29:59 +10:00
webseed WebSeed PathEscaper API tweaks 2022-04-26 10:46:01 +10:00
webtorrent Close data channel on offer init errors 2022-07-12 16:15:51 +10:00
.deepsource.toml Update .deepsource.toml 2022-02-21 21:23:20 +00:00
.gitignore Ignore request strategy run gobs and envrcs 2021-10-19 14:08:56 +11:00
.golangci.yml Update staticcheck go target to 1.16 2021-08-19 16:06:24 +10:00
LICENSE Update LICENSE 2018-01-27 11:21:33 +11:00
NOTES.md Add NOTES 2021-09-20 12:31:44 +10:00
README.md Add exatorrent and finish tidying downstream projects 2022-06-25 17:28:23 +10:00
SECURITY.md Create SECURITY.md 2021-05-14 10:17:19 +10:00
TODO Update TODO file 2018-04-14 21:44:50 +10:00
bad_storage.go Rework storage.TorrentImpl to support shared capacity key 2021-06-07 13:01:39 +10:00
bep40.go Move IpPort to missinggo 2018-11-16 10:35:30 +11:00
bep40_test.go Use webrtc local addr for webrtc conn peer priority 2022-07-12 15:52:03 +10:00
callbacks.go Expose more callbacks and Request and ChunkSpec 2021-01-28 14:23:22 +11:00
client-nowasm_test.go Revert "Remove old-style build tags" 2021-11-01 11:26:58 +11:00
client.go pass TrackerDialContext to webtorrent trackers (#785) 2022-11-14 19:20:08 +11:00
client_test.go [client] Handle nil IP in badPeerAddr() 2022-04-26 10:59:12 +10:00
config.go support providing a DialContext for the http client (#786) 2022-11-14 19:19:02 +11:00
conn_stats.go Track chunks read for webseed peers too 2021-06-07 13:01:40 +10:00
deferrwl.go lockWithDeferreds: inlineable Unlock() (#603) 2021-09-17 13:08:00 +10:00
dht.go Don't wait for DHT pings 2021-12-23 14:00:00 +11:00
dialer.go Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
doc.go BEP27 isn't implemented 2020-05-24 17:47:16 +10:00
example_test.go Add File.NewReader 2018-01-06 16:37:13 +11:00
file.go use rLock where can, part2 (#767) 2022-07-13 20:04:03 +10:00
file_test.go gofumpt 2021-11-08 14:47:01 +11:00
global.go go fmt ./... 2021-10-05 17:48:52 +11:00
go.mod Update github.com/anacrolix/dht/v2 2022-09-18 11:26:58 +10:00
go.sum Update github.com/anacrolix/dht/v2 2022-09-18 11:26:58 +10:00
handshake.go Add the ReceiveEncryptedHandshakeSkeys callback 2020-11-05 13:28:45 +11:00
handshake_test.go Support AllowedFast and enable fast extension 2018-02-05 00:18:38 +11:00
ipport.go Possible fix for GitHub code scanning warning 2021-05-14 15:43:13 +10:00
issue97_test.go test: use `T.TempDir` to create temporary test directory (#718) 2022-02-02 18:26:46 +11:00
issue211_test.go Fix leaked mmap storage in test 2021-12-16 14:19:21 +11:00
listen.go Optimize `LoopbackListenHost()` (#615) 2021-09-14 21:54:37 +10:00
main_test.go Replace internal/tmproot with Testing.TB.TempDir 2021-02-22 14:30:09 +11:00
misc.go Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
misc_test.go Drop bradfitz/iter dependency (#605) 2021-09-14 13:46:50 +10:00
network_test.go Pass client logger to anacrolix/go-libutp sockets (#722) 2022-02-15 16:18:32 +11:00
networks.go Improve network handling and only listen networks we will use 2018-11-28 10:30:21 +11:00
ordered-bitmap.go Retain peer local request ordering 2022-05-09 12:05:50 +10:00
otel.go webtorrent: Create data channel earlier per webrtc examples 2022-07-12 16:15:50 +10:00
peer-conn-msg-writer.go use rLock where can, part2 (#767) 2022-07-13 20:04:03 +10:00
peer-impl.go Support banning webseeds 2022-03-11 14:05:31 +11:00
peer.go Use webrtc local addr for webrtc conn peer priority 2022-07-12 15:52:03 +10:00
peer_info.go Don't expose Peer network through RemoteAddr 2021-01-25 15:49:48 +11:00
peer_infos.go Rename Peer to PeerInfo, and unexport PeerInfos 2020-05-29 19:44:48 +10:00
peerconn.go use rLock where can, part2 (#767) 2022-07-13 20:04:03 +10:00
peerconn_test.go Use webrtc local addr for webrtc conn peer priority 2022-07-12 15:52:03 +10:00
peerid.go Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
peerid_test.go Change a bunch of Stringer and status stuff 2018-02-05 17:47:47 +11:00
pending-requests.go Revert "Use a flat slice for pending request counts" 2021-11-25 22:55:02 +11:00
pending-requests_test.go Comment out pending requests tests and asserts 2021-11-25 22:59:20 +11:00
pex.go PEX: remove workaround for krpc marshaller 2021-12-11 19:44:42 +00:00
pex_test.go PEX: fluid event log 2021-12-12 00:27:28 +11:00
pexconn.go PEX: fluid event log 2021-12-12 00:27:28 +11:00
pexconn_test.go Use webrtc local addr for webrtc conn peer priority 2022-07-12 15:52:03 +10:00
piece.go Do torrent storage flush on piece completion (#755) 2022-07-07 15:46:27 +10:00
piecestate.go Track piece marking state separately 2020-11-21 13:44:23 +11:00
portfwd.go Big logging cleanup to improve experience from README 2021-05-04 12:44:51 +10:00
prioritized_peers.go Rename Peer to PeerInfo, and unexport PeerInfos 2020-05-29 19:44:48 +10:00
prioritized_peers_test.go Rename Peer to PeerInfo, and unexport PeerInfos 2020-05-29 19:44:48 +10:00
protocol.go Expose more callbacks and Request and ChunkSpec 2021-01-28 14:23:22 +11:00
ratelimitreader.go Fix download rate limiter and add test 2018-01-28 15:28:45 +11:00
reader.go Fix races around Reader and Torrent.Drop when Torrent is closed 2022-04-11 15:44:24 +10:00
reader_test.go Replace internal/tmproot with Testing.TB.TempDir 2021-02-22 14:30:09 +11:00
request-strategy-impls.go Add and use typed roaring bitmap 2022-05-09 12:05:50 +10:00
requesting.go Use reusable roaring iterators 2022-05-12 10:42:16 +10:00
requesting_test.go Shuffle duplicate requests 2021-12-12 16:56:00 +11:00
rlreader_test.go Drop bradfitz/iter dependency (#605) 2021-09-14 13:46:50 +10:00
roaring.go Add and use typed roaring bitmap 2022-05-09 12:05:50 +10:00
smartban.go Switch to external anacrolix/generics 2022-01-22 18:43:04 +11:00
socket.go Move a bunch of stuff into subpackages 2022-11-15 23:31:26 +11:00
sources.go Export Torrent.UseSources 2022-03-15 13:42:00 +11:00
spec.go WebSeed PathEscaper API tweaks 2022-04-26 10:46:01 +10:00
struct_test.go Export Peer 2021-01-25 15:49:48 +11:00
t.go use rLock where can, part2 (#767) 2022-07-13 20:04:03 +10:00
test_test.go Add TestHaveAllThenBitfield 2021-12-23 14:00:00 +11:00
testing.go Make use of the lazylog branch of anacrolix/log 2022-01-31 17:22:36 +11:00
torrent-piece-request-order.go Switch to ajwerner/btree in piece request ordering 2022-01-22 18:40:33 +11:00
torrent.go Win limit amount of peers per udp packet (#771) 2022-08-08 15:38:19 +10:00
torrent_mmap_test.go Revert "Remove old-style build tags" 2021-11-01 11:26:58 +11:00
torrent_stats.go Add TorrentStats.PiecesComplete 2021-10-07 13:50:13 +11:00
torrent_test.go Add TestHaveAllThenBitfield 2021-12-23 14:00:00 +11:00
tracker_scraper.go Add ability to set DialContext/ListenPacket for tracker announcements (#760) 2022-07-07 15:51:58 +10:00
undirtied-chunks-iter.go Use reusable roaring iterators 2022-05-12 10:42:16 +10:00
undirtied-chunks-iter_test.go Use reusable roaring iterators 2022-05-12 10:42:16 +10:00
url-net-addr.go Generate a RemoteAddr for webseed Peers 2021-01-25 15:49:48 +11:00
utp.go Add support for disabling IPv4 and IPv4 peers 2018-02-16 10:46:11 +11:00
utp_go.go Pass client logger to anacrolix/go-libutp sockets (#722) 2022-02-15 16:18:32 +11:00
utp_libutp.go Pass client logger to anacrolix/go-libutp sockets (#722) 2022-02-15 16:18:32 +11:00
utp_test.go Pass client logger to anacrolix/go-libutp sockets (#722) 2022-02-15 16:18:32 +11:00
webrtc.go Trace some webrtcNetConn use 2022-07-12 16:15:50 +10:00
webseed-peer.go Add and use typed roaring bitmap 2022-05-09 12:05:50 +10:00
worse-conns.go Optimize the use of worstConnSlice again 2021-11-29 10:07:48 +11:00
worse-conns_test.go Optimize the use of worstConnSlice again 2021-11-29 10:07:48 +11:00
wstracker.go pass TrackerDialContext to webtorrent trackers (#785) 2022-11-14 19:20:08 +11:00

README.md

torrent

PkgGoDev

This repository implements BitTorrent-related packages and command-line utilities in Go. The emphasis is on use as a library from other projects. It's been used 24/7 in production by downstream services since late 2014. The implementation was specifically created to explore Go's concurrency capabilities, and to include the ability to stream data directly from the BitTorrent network. To this end it supports seeking, readaheads and other features exposing torrents and their files with the various Go idiomatic io package interfaces. This is also demonstrated through torrentfs.

There is support for protocol encryption, DHT, PEX, uTP, and various extensions. There are several data storage backends provided: blob, file, bolt, mmap, and sqlite, to name a few. You can write your own to store data for example on S3, or in a database.

Some noteworthy package dependencies that can be used for other purposes include:

Installation

Install the library package with go get github.com/anacrolix/torrent, or the provided cmds with go install github.com/anacrolix/torrent/cmd/...@latest.

Library examples

There are some small examples in the package documentation.

Mentions

Downstream projects

There are several web-frontends, sites, Android clients and supporting services among the known public projects:

Help

Communication about the project is primarily through Discussions and the issue tracker.

Command packages

Here I'll describe what some of the packages in ./cmd do. See installation to make them available.

torrent

torrent download

Downloads torrents from the command-line.

$ torrent download 'magnet:?xt=urn:btih:KRWPCX3SJUM4IMM4YF5RPHL6ANPYTQPU'
... lots of jibba jabber ...
downloading "ubuntu-14.04.2-desktop-amd64.iso": 1.0 GB/1.0 GB, 1989/1992 pieces completed (1 partial)
2015/04/01 02:08:20 main.go:137: downloaded ALL the torrents
$ md5sum ubuntu-14.04.2-desktop-amd64.iso
1b305d585b1918f297164add46784116  ubuntu-14.04.2-desktop-amd64.iso
$ echo such amaze
wow

torrent metainfo magnet

Creates a magnet link from a torrent file. Note the extracted trackers, display name, and info hash.

$ torrent metainfo testdata/debian-10.8.0-amd64-netinst.iso.torrent magnet
magnet:?xt=urn:btih:4090c3c2a394a49974dfbbf2ce7ad0db3cdeddd7&dn=debian-10.8.0-amd64-netinst.iso&tr=http%3A%2F%2Fbttracker.debian.org%3A6969%2Fannounce

See torrent metainfo --help for other metainfo related commands.

torrentfs

torrentfs mounts a FUSE filesystem at -mountDir. The contents are the torrents described by the torrent files and magnet links at -metainfoDir. Data for read requests is fetched only as required from the torrent network, and stored at -downloadDir.

$ mkdir mnt torrents
$ torrentfs -mountDir=mnt -metainfoDir=torrents &
$ cd torrents
$ wget http://releases.ubuntu.com/14.04.2/ubuntu-14.04.2-desktop-amd64.iso.torrent
$ cd ..
$ ls mnt
ubuntu-14.04.2-desktop-amd64.iso
$ pv mnt/ubuntu-14.04.2-desktop-amd64.iso | md5sum
996MB 0:04:40 [3.55MB/s] [========================================>] 100%
1b305d585b1918f297164add46784116  -