Ray ClusterをKubernetes上で試す

Rayとは

Pythonを簡単にスケールできるようにするオープンソースプロジェクト

• Deep Learning
• Reinforcement Learning
• Hyperparameter Tuning
• General Python apps
• Model Serving
• Data Processing

MLがメインのようにみえるが、様々な用途に使える。

Ray-operator

RayをKubernetes上で動かすためのOperatorがある

Custom Resource: RayCluster

ステップ

インストール

kubectl apply -k "github.com/ray-project/kuberay/ray-operator/config/default"

Rayクラスタ作成

RayClusterの作成

kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.heterogeneous.yaml

ConfigMapの作成

kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/config-map-ray-code.yaml

ConfigMapに、Pythonのコードが入っている

apiVersion: v1
kind: ConfigMap
metadata:
  name: ray-code
data: 
  sample_code.py: |
    import ray
    from os import environ
    redis_pass = environ.get("REDIS_PASSWORD") 
    print("trying to connect to Ray!")
    ray.init(address="auto", _redis_password=redis_pass)
    print("now executing some code with Ray!")
    import time
    start = time.time()
    @ray.remote
    def f():
      time.sleep(0.01)
      return ray._private.services.get_node_ip_address()
    values=set(ray.get([f.remote() for _ in range(1000)]))
    print("Ray Nodes: ",str(values))
    file = open("/tmp/ray_nodes.txt","a")
    file.write("available nodes: %s\n" % str(values))
    file.close()
    end = time.time()
    print("Execution time = ",end - start)

確認: head 1つとworkerが4つ動いている

kubectl get pod
NAME                                                 READY   STATUS    RESTARTS   AGE
raycluster-heterogeneous-head-72l7l                  1/1     Running   0          2m31s
raycluster-heterogeneous-worker-medium-group-zcml2   1/1     Running   0          2m31s
raycluster-heterogeneous-worker-small-group-fvxd5    1/1     Running   0          2m31s
raycluster-heterogeneous-worker-small-group-hrlt7    1/1     Running   0          2m31s
raycluster-heterogeneous-worker-small-group-nkqj8    1/1     Running   0          2m31s

参考

• https://towardsdatascience.com/modern-parallel-and-distributed-python-a-quick-tutorial-on-ray-99f8d70369b8
• https://github.com/ray-project/kuberay
• https://ray-project.github.io/2018/07/15/parameter-server-in-fifteen-lines.html
• https://github.com/apache/arrow/blob/master/cpp/apidoc/tutorials/plasma.md#:~:text=Plasma%20clients%20are%20processes%20that,once%20they%20have%20been%20created.