I am Ryo Kanbayashi (ryo_grid).
This post introduces new software I developed.
What is this?
- a tool transfer files and connect pipe between PCs in different NATs, etc
- It is a feeling inspired by Piping Server
- using the data channel functionality of WebRTC
- with UDP hole punching, my tool climbs over NAT
- the P2P communication path is SCTP over DTLS over UDP (it is a reliable communication path and is also an encrypted communication path)
Architecture & Implementation
- basically the same as WebRTC
- for getting information for NAT traversal (UDP hole punching), WebRTC client uses the STUN server (ICE mechanism)
- STUN server which Google is offering for testing is used
- For WebRTC working, a server which is called signaling server is needed to mediate machines (two machines want to do p2p communication) in addition to STUN server
- so, I am offering a singaling server which can be shared by multiple users
- my signaling server's address is wss://ryogrid.net:11985/ . it uses WebSocket protcol over SSL
- The communication between the user and the signaling server is encrypted
- implementation language is Python. The part of the data channel construction of WebRTC uses a library called aiortc
Requirement to use
- OS has been confirmed to work on Windows, Mac, Linux
- Python runtime version 3.7+ is needed. if not installed, please install with considering using pyenv etc.
Attention (1)
- UDP hole punching may not be possible depending on NAT specifications, functions, settings and network configuration, so it's not always possible to establish p2p commucation path
- but about 80% works well I think
- signaling server is basically no more than a bridge of information to set up a communication path, but if I
tried, I can see the contents of information for communication path establishment (info like key for encryption is not included!)- signaling server does not relay much secret information. so it's meaningless if I look at
it - I can only say that it is an extreme theory, please believe me :)
- About information of me is as follows
- signaling server does not relay much secret information. so it's meaningless if I look at
Installationn (pip module)
$ pip install onatlib
if you install with pip module, description of my server and tool exection below are replaced to ...
- python p2p_com_local_server.py [arguments] -> p2p_com_local_serv [arguments]
- python pync.py [arguments] -> pync [arguments]
- python shareable_ws_signailing_serv.py [arguments] -> shareable_ws_signaling_serv [arguments]
Installation (from github)
clone repository
$ git clone -b first-release-0624 https://github.com/ryogrid/Over-NAT-Lib.git
install required pip modues
$ cd Over-NAT-Lib
$ pip install -r requirements.txt
Using（preparation part）
First, move to the tools directory of the repository you cloned
$ cd Over-Nat-Lib/tools
- Signaling Server match users by receiving identifier which is shared between two machines (persons)
- the identifier should be alphanumeric character string and length of it should be longer than 9 characters
- it must not able to be guess by other persons for security
- Therefore, First, please decide identifier string and share between persons who want to p2p communicate
- due to my implementation reason, in order to identify which of the two, each person select "bob" or "tom"
- there is no particular reason about selected names (tom and bob)
Attention (2)
- on Windows, please use the command prompt. In the case of Power Shell, the handling of pipes and stdout is special, and it does not work well with my tools
- from the characteristics of the established communication path, the local server on the sending side buffering 1 MB of data and do not send data to the remote until buffered size reaches 1MB (If the EOF is detected even if 1 MB is not reached, the buffered data will be sent regardless of the size) for communication performance(=throughput)
Using (channel establishment)
- current directory is in the tools directory
- executing my tool (p2P_com_local_server.py) like below (left of this post) spawns p2P_com_local_server.py process with diffent parameters. as a result, 3 processes starts.
- If the parent is terminated with Ctrl-C, the other two processes are also terminated, but if the parent dies for some reason and can not perform termination processing, the two child processes remain as they are. So, if that case occurs, please kill these with your own....
first, establish a communication channel. The program which is executed here is like a server program that runs on the local host.
the established communication path is designed to be maintained by this server program and to be used by other programs.
for example, the identifier is "d5ocew761d" on below.
Note that below command execution redirecting stderr to a file, but this is because I output a not little amount of debugging output to stderr. it is not necessary.
if you want to see the detailed debug output for establishing a communication path (output by the library which is used internally), it is interesting to add the -v option.
Tom side
$ python p2p_com_local_server.py --signaling share-websocket --signaling-host ryogrid.net --signaling-port 11985 --name tom --secure-signaling d5ocew761d 2> hogehoge.txt
Bob side
$ python p2p_com_local_server.py --signaling share-websocket --signaling-host ryogrid.net --signaling-port 11985 --name bob --secure-signaling d5ocew761d 2> hogehoge.txt
execution order can be reversed.
communication path will be established in as little as 3 minutes after both start up.
You can confirm that it is established if the following is outputted.
sender_proc: datachannel established
recv_proc: datachannel established
if NAT traversal fails, output like below are shown, but if you wait for 10 minutes, you can think that the traversal fails.
hole punching to remote machine failed.
if communication channel is established, in each of the host, the write port of the communication channel
opens at 10100. read port at 10200.
If you want to try it on a single PC, please use --slide-stream-ports option at second execution, which changes write port to 10101 and read port to 10201.
If you want to quit, please type Ctrl-c in the terminal which my local server was executed.
Using (pipe transfer)
- following assumes that the communication channel has been established via pipe.
- pync.py which is like Netcat(nc) tool is used
- this is a example that uses pipe stdin and redirect stdout to remote machine
- current directory is tools directory
The sender
send contens of sometexts.txt to remote.
$ cat sometexts.txt | python pync.py -c -p 10100
on windows, you can use type command instead of cat command
The receiver
count the number of received data lines.
$ python pync.py -r -p 10200 | wc -l
Note that the order of execution does not matter.
Transfer end
- transfer is completed, pync.py ends
- the size of the receive buffer at the socket (write port) of the local server is not small. so, if the data to be transferred is about several KB in size, the sending side pync.py may finish sending (to the local server) and exit. though your receiver side pync.py is connected yet
- even if data transfer finished, established communication channel is not disconnected. so, for example, you can do same transfer like above multi times
Using (direct file transfer)
- following assumes that the already communication channel has been established.
- current directory is tools directory
- this is a feature that file transfer without the receiving side person approve
- transferred files are basically written to the current directory at the time of execution of p2p_com_local_server.py on the receiving side, but does NOT if the file name is specified with malicious intent (check mechanism is not implemented yet...)
- the difference with pipe transfer example already explained is that pync.py is given the -f option to specify the file name at the write destination
File sender side
file name is somefile.tar.gz. Here, the same name as the file which is cated & piped.
but it does not matter if different file name is specified as argument of pync.py, the filename specified to pync.py is used of file cration for write transferred data.
$ cat somefile.tar.gz | python pync.py -f somefile.tar.gz -c -p 10100
receiver does not need to do anything.
pync.py exits when the transfer is complete.
Library for using established transport
- I wrote simple library
- it enable you to access local server's transport transparently
- receiver side can use data receive callback
- library returns pure python socket object
- please use below example code as a refecence (simple echo server and client)
Setup own signaling server
- if you can't trust me, you can set up your own signaling server.
- current directory is tools directory
With encryption
- place certificate files to tools directory
- privkey.pem and fullchain.pem
start server.
$ python python shareable_ws_signaling_serv.py --secure --port <port to listen> &> sigserv_log.txt
server exit can be with typing Ctrl-c in the terminal which my local server was executed.
the local server (p2p_com_local_server.py) for data transfer as described above can be executed with the --secure-signaling option.
With no encryption
- current directory is tools directory
start server.
$ python python shareable_ws_signaling_serv.py --port <port to listen> &> sigserv_log.txt
server exit can be with typing Ctrl-c in the terminal which my local server was executed.
the local server (p2p_com_local_server.py) for data transfer as described above can not be executed with the --secure-signaling option.
Binary (execution file format) distribution (Windows platform only)
I created Binary distribution of tools of Over-NAT-Lib.
These works without python environment.
Have Fun!