PyGithubの認証で注意すべきこと

はじめに

PyGithubを利用して複数ユーザのリソースにアクセスしようとしたら、必ず同じユーザのリソースにしかアクセスできず途方にくれた。
認証にはユーザ毎に生成可能なPrivateAccessTokenを利用していたが、何度設定し直しても状況が変わらなかったため、仕方なくソースを読むことにした。
読み込むのに時間がかかったので、作業を無駄にしないためにもメモしておく。

結論から言えば.netrcファイルがホームディレクトリに配置されており、それが最優先されていたことが原因だった。

以前PyGithubと同じようなRubyのOctkitというツールを利用してGithubのAPIを利用した事があったが、おそらくそのタイミングで.netrcファイルを作ったのだろう。
そんなことをすっかり忘れ、しかもその設定が最優先されるなんて思いもよらなかった。当時の自分になにやってんだと言いたい。

実行環境

というわけで調査を開始したわけだが、その前にまず実行環境の確認から。

$ python --version
Python 3.6.5

$ pip list
Package    Version
---------- --------
certifi    2019.3.9
chardet    3.0.4
Deprecated 1.2.5
idna       2.8
pip        19.0.3
PyGithub   1.43.5
PyJWT      1.7.1
requests   2.21.0
setuptools 39.0.1
urllib3    1.24.1
wrapt      1.11.1

調査

はじめに

今回調査に時間がかかってしまった理由は「実際にどんなリクエストが送信されているか」がすぐに分からなかったから。

Getting redirected repo does not work if debug logging is enabled. · Issue #470 · PyGithub/PyGithub · GitHub

こちらを参考にDEBUGレベルでログを出力しても、肝心のトークンはセキュリティを気にして表示してくれない。
当たり前のような機能だけど、今回に限ってはお節介も甚だしい。（完全に八つ当たりです）

実行手順

以下の要領で実行。
※enndpoint0-2は今後の説明のための実行の起点。

$ python
Python 3.6.5 (default, Dec  7 2018, 10:51:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>>
>>> from github import Github
>>>
>>> g = Github('<MY_PRIVATE_ACCESS_TOKEN>') #★endpoint0
>>> u = g.get_user()                        #★endpoint1
>>> u.login                                 #★endpoint2
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443
DEBUG:urllib3.connectionpool:https://api.github.com:443 "GET /user HTTP/1.1" 200 None
DEBUG:github.Requester:GET https://api.github.com/user {'Authorization': 'token (oauth token removed)', 'User-Agent': 'PyGithub/Python'} None ==> 200 {'date': 'Mon, 01 Apr 2019 12:32:28 GMT', 'content-type': 'application/json; charset=utf-8', 'transfer-encoding': 'chunked', 'server': 'GitHub.com', 'status': '200 OK', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '4999', 'x-ratelimit-reset': '1554125548', 'cache-control': 'private, max-age=60, s-maxage=60', 'vary': 'Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding', 'etag': 'W/"97ff3b6da70a11af51d10e69099f2a96"', 'last-modified': 'Mon, 01 Apr 2019 03:06:05 GMT', 'x-oauth-scopes': 'admin:repo_hook, repo', 'x-accepted-oauth-scopes': '', 'x-github-media-type': 'github.v3; format=json', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'content-encoding': 'gzip', 'x-github-request-id': 'E056:8730:37E5769:43FEE21:5CA204DC'} {"login":"<ログインユーザ名>","id":<ユーザID:整数値>,"node_id":"<ノードID>","avatar_url":"https://avatars3.githubusercontent.com/u/<ユーザID:整数値>?v=4","gravatar_id":"","url":"https://api.github.com/users/<ログインユーザ名>","html_url":"https://github.com/<ログインユーザ名>","followers_url":"https://api.github.com/users/<ログインユーザ名>/followers","following_url":"https://api.github.com/users/<ログインユーザ名>/following{/other_user}","gists_url":"https://api.github.com/users/<ログインユーザ名>/gists{/gist_id}","starred_url":"https://api.github.com/users/<ログインユーザ名>/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/<ログインユーザ名>/subscriptions","organizations_url":"https://api.github.com/users/<ログインユーザ名>/orgs","repos_url":"https://api.github.com/users/<ログインユーザ名>/repos","events_url":"https://api.github.com/users/<ログインユーザ名>/events{/privacy}","received_events_url":"https://api.github.com/users/<ログインユーザ名>/received_events","type":"User","site_admin":false,"name":null,"company": null,"blog": null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":0,"public_gists":0,"followers":3,"following":0,"created_at":"2013-03-05T15:00:18Z","updated_at":"2019-04-01T03:06:05Z"}
'<ログインユーザ名>'

<MY_PRIVATE_ACCESS_TOKEN> がリソースを操作したいユーザのPrivateAccessToken。
本来なら <ログインユーザ名> にはこのアクセストークンに対応するユーザ名が表示されるのだが、実際には別のユーザの情報が表示された。

しかたなく、モジュールにprint文を挟んでデバッグすることに。

以下モジュールのソースコードを読んだ流れを記載する。

ソースコード探訪

実行手順のendpoint0-2から始まり、枝番の順に目を通した。

そして、最終的にはnetrcの設定がある場合に、RequestHeaderが上書きされている事がわかった。

lib/python3.6/site-packages/github/MainClass.py

 93 class Github(object): ★endpoint0-1
 94     """
 95     This is the main class you instantiate to access the Github API v3. Optional parameters allow different authentication methods.
 96     """
 97
 98     def __init__(self, login_or_token=None, password=None, jwt=None, base_url=DEFAULT_BASE_URL, timeout=DEFAULT_TIMEOUT, client_id=None, client_secret=None, user_agent='PyGithub/Python', per_page=DEFAULT_PER_PAGE, api_preview=False, verify=True): ★endpoint0-2
 99         """
100         :param login_or_token: string
101         :param password: string
102         :param base_url: string
103         :param timeout: integer
104         :param client_id: string
105         :param client_secret: string
106         :param user_agent: string
107         :param per_page: int
108         :param verify: boolean or string
109         """
110
111         assert login_or_token is None or isinstance(login_or_token, str), login_or_token
112         assert password is None or isinstance(password, str), password
113         assert jwt is None or isinstance(jwt, str), jwt
114         assert isinstance(base_url, str), base_url
115         assert isinstance(timeout, int), timeout
116         assert client_id is None or isinstance(client_id, str), client_id
117         assert client_secret is None or isinstance(client_secret, str), client_secret
118         assert user_agent is None or isinstance(user_agent, str), user_agent
119         assert isinstance(api_preview, (bool))
120         self.__requester = Requester(login_or_token, password, jwt, base_url, timeout, client_id, client_secret, user_agent, per_page, api_preview, verify) ★endpoint0-3
…
220     def get_user(self, login=github.GithubObject.NotSet): ★endpoint1-1
221         """
222         :calls: `GET /users/:user <http://developer.github.com/v3/users>`_ or `GET /user <http://developer.github.com/v3/users>`_
223         :param login: string
224         :rtype: :class:`github.NamedUser.NamedUser`
225         """
226         assert login is github.GithubObject.NotSet or isinstance(login, str), login
227         if login is github.GithubObject.NotSet:
228             return AuthenticatedUser.AuthenticatedUser(self.__requester, {}, {"url": "/user"}, completed=False) ★endpoint1-2
229         else:
230             headers, data = self.__requester.requestJsonAndCheck(
231                 "GET",
232                 "/users/" + login
233             )
234             return github.NamedUser.NamedUser(self.__requester, headers, data, completed=True)

lib/python3.6/site-packages/github/AuthenticatedUser.py

  63 class AuthenticatedUser(github.GithubObject.CompletableGithubObject): ★endpoint1-3
  64     """
  65     This class represents AuthenticatedUsers as returned by https://developer.github.com/v3/users/#get-the-authenticated-user
  66
  67     An AuthenticatedUser object can be created by calling ``get_user()`` on a Github object.
  68     """
  69
  70     def __repr__(self):
  71         return self.get__repr__({"login": self._login.value})
   …
 225     @property
 226     def login(self): ★endpoint2-1
 227         """
 228         :type: string
 229         """
 230         self._completeIfNotSet(self._login) ★endpoint2-2
 231         return self._login.value

lib/python3.6/site-packages/github/GithubObject.py

 69 class GithubObject(object):  ★endpoint1-6
 70     """
 71     Base class for all classes representing objects returned by the API.
 72     """
 73
 74     '''
 75     A global debug flag to enable header validation by requester for all objects
 76     '''
 77     CHECK_AFTER_INIT_FLAG = False
 78
 79     @classmethod
 80     def setCheckAfterInitFlag(cls, flag):
 81         cls.CHECK_AFTER_INIT_FLAG = flag
 82
 83     def __init__(self, requester, headers, attributes, completed):  ★endpoint1-7
 84         self._requester = requester
 85         self._initAttributes()
 86         self._storeAndUseAttributes(headers, attributes)
 87
 88         # Ask requester to do some checking, for debug and test purpose
 89         # Since it's most handy to access and kinda all-knowing
 90         if self.CHECK_AFTER_INIT_FLAG:  # pragma no branch (Flag always set in tests)
 91             requester.check_me(self)
  …
250 class CompletableGithubObject(GithubObject): ★endpoint1-4
251     def __init__(self, requester, headers, attributes, completed):
252         GithubObject.__init__(self, requester, headers, attributes, completed) ★endpoint1-5
253         self.__completed = completed
  …
261     def _completeIfNotSet(self, value):  ★endpoint2-3
262         if value is NotSet:
263             self._completeIfNeeded() ★endpoint2-4
264
265     def _completeIfNeeded(self):  ★endpoint2-5
266         if not self.__completed:
267             self.__complete() ★endpoint2-6
268
269     def __complete(self):  ★endpoint2-7
270         headers, data = self._requester.requestJsonAndCheck(
271             "GET",
272             self._url.value
273         ) ★endpoint2-8
274         self._storeAndUseAttributes(headers, data)
275         self.__completed = True

lib/python3.6/site-packages/github/Requester.py

 90 class HTTPSRequestsConnectionClass(object):
 91     # mimic the httplib connection object
 92     def __init__(self, host, port=None, strict=False, timeout=None, **kwargs):
 93         self.port = port if port else 443
 94         self.host = host
 95         self.protocol = "https"
 96         self.timeout = timeout
 97         self.verify = kwargs.get("verify", True)
 98         self.session = requests.Session()
 99
100     def request(self, verb, url, input, headers):
101         self.verb = verb
102         self.url = url
103         self.input = input
104         self.headers = headers
105
106     def getresponse(self):  ★endpoint2-19
107         verb = getattr(self.session, self.verb.lower()) ★ ★endpoint2-20: Sessionクラスのgetメソッドを取得
108         url = "%s://%s:%s%s" % (self.protocol, self.host, self.port, self.url)
109         r = verb(url, headers=self.headers, data=self.input, timeout=self.timeout, verify=self.verify, allow_redirects=False) ★endpoint2-21: getメソッド実行
110         return RequestsResponse(r)
111
112     def close(self):
113         return
…
142 class Requester: ★endpoint0-4
143     __httpConnectionClass = HTTPRequestsConnectionClass
144     __httpsConnectionClass = HTTPSRequestsConnectionClass ★endpoint0-6(last)
…
217     def __init__(self, login_or_token, password, jwt, base_url, timeout, client_id, client_secret, user_agent, per_page, api_preview, verify):
218         self._initializeDebugFeature()
219
220         if password is not None:
221             login = login_or_token
222             if atLeastPython3:
223                 self.__authorizationHeader = "Basic " + base64.b64encode((login + ":" + password).encode("utf-8")).decode("utf-8").replace('\n', '')  # pragma no cover (Covered by Authentication.testAuthorizationHeaderWithXxx with Python 3)
224             else:
225                 self.__authorizationHeader = "Basic " + base64.b64encode(login + ":" + password).replace('\n', '')
226         elif login_or_token is not None:
227             token = login_or_token
228             self.__authorizationHeader = "token " + token
229         elif jwt is not None:
230             self.__authorizationHeader = "Bearer " + jwt
231         else:
232             self.__authorizationHeader = None
233
234         self.__base_url = base_url
235         o = urllib.parse.urlparse(base_url)
236         self.__hostname = o.hostname
237         self.__port = o.port
238         self.__prefix = o.path
239         self.__timeout = timeout
240         self.__scheme = o.scheme
241         if o.scheme == "https":
242             self.__connectionClass = self.__httpsConnectionClass ★endpoint0-5
243         elif o.scheme == "http":
244             self.__connectionClass = self.__httpConnectionClass
245         else:
246             assert False, "Unknown URL scheme"
…
263     def requestJsonAndCheck(self, verb, url, parameters=None, headers=None, input=None):  ★endpoint2-9
264         return self.__check(*self.requestJson(verb, url, parameters, headers, input, self.__customConnection(url))) ★endpoint2-10
…
320     def requestJson(self, verb, url, parameters=None, headers=None, input=None, cnx=None): ★endpoint2-11
321         def encode(input):
322             return "application/json", json.dumps(input)
323
324         return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode) ★endpoint2-12
…
356     def __requestEncode(self, cnx, verb, url, parameters, requestHeaders, input, encode): ★endpoint2-13
357         assert verb in ["HEAD", "GET", "POST", "PATCH", "PUT", "DELETE"]
…
376
377         status, responseHeaders, output = self.__requestRaw(cnx, verb, url, requestHeaders, encoded_input) ★endpoint2-14
…
391     def __requestRaw(self, cnx, verb, url, requestHeaders, input): ★endpoint2-15
392         original_cnx = cnx
393         if cnx is None:
394             cnx = self.__createConnection() ★endpoint2-16
395         cnx.request(
396             verb,
397             url,
398             input,
399             requestHeaders
400         )
401         response = cnx.getresponse() ★endpoint2-20
…
452     def __createConnection(self): ★endpoint2-17
453         kwds = {}
454         if not atLeastPython3:  # pragma no branch (Branch useful only with Python 3)
455             kwds["strict"] = True  # Useless in Python3, would generate a deprecation warning
456         kwds["timeout"] = self.__timeout
457         kwds["verify"] = self.__verify
458
459         if self.__persist and self.__connection is not None:
460             return self.__connection
461
462         self.__connection = self.__connectionClass(self.__hostname, self.__port, **kwds) ★endpoint2-18(endpoint0-5で設定したもの)
463
464         return self.__connection ★endpoint2-19

lib/python3.6/site-packages/requests/sessions.py

340 class Session(SessionRedirectMixin):
…
426     def prepare_request(self, request):  ★endpoint2-26
427         """Constructs a :class:`PreparedRequest <PreparedRequest>` for
428         transmission and returns it. The :class:`PreparedRequest` has settings
429         merged from the :class:`Request <Request>` instance and those of the
430         :class:`Session`.
431
432         :param request: :class:`Request` instance to prepare with this
433             session's settings.
434         :rtype: requests.PreparedRequest
435         """
436         cookies = request.cookies or {}
437
438         # Bootstrap CookieJar.
439         if not isinstance(cookies, cookielib.CookieJar):
440             cookies = cookiejar_from_dict(cookies)
441
442         # Merge with session cookies
443         merged_cookies = merge_cookies(
444             merge_cookies(RequestsCookieJar(), self.cookies), cookies)
445
446         # Set environment's basic authentication if not explicitly set.
447         auth = request.auth
448         if self.trust_env and not auth and not self.auth:
449             auth = get_netrc_auth(request.url) ★endpoint2-27: ここでnetrcの設定を取得
450
451         p = PreparedRequest()
452         p.prepare(
453             method=request.method.upper(),
454             url=request.url,
455             files=request.files,
456             data=request.data,
457             json=request.json,
458             headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
459             params=merge_setting(request.params, self.params),
460             auth=merge_setting(auth, self.auth),
461             cookies=merged_cookies,
462             hooks=merge_hooks(request.hooks, self.hooks),
463         ) ★endpoint2-28
464         return p
465
466     def request(self, method, url,
467             params=None, data=None, headers=None, cookies=None, files=None,
468             auth=None, timeout=None, allow_redirects=True, proxies=None,
469             hooks=None, stream=None, verify=None, cert=None, json=None): ★endpoint2-24
…
506         # Create the Request.
507         req = Request(
508             method=method.upper(),
509             url=url,
510             headers=headers,
511             files=files,
512             data=data or {},
513             json=json,
514             params=params or {},
515             auth=auth,
516             cookies=cookies,
517             hooks=hooks,
518         )
519         prep = self.prepare_request(req) ★endpoint2-25
…
537     def get(self, url, **kwargs):  ★endpoint2-22
538         r"""Sends a GET request. Returns :class:`Response` object.
539
540         :param url: URL for the new :class:`Request` object.
541         :param \*\*kwargs: Optional arguments that ``request`` takes.
542         :rtype: requests.Response
543         """
544
545         kwargs.setdefault('allow_redirects', True)
546         return self.request('GET', url, **kwargs) ★endpoint2-23

lib/python3.6/site-packages/requests/models.py

272 class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
…
307     def prepare(self,
308             method=None, url=None, headers=None, files=None, data=None,
309             params=None, auth=None, cookies=None, hooks=None, json=None):  ★endpoint2-29
310         """Prepares the entire request with the given parameters."""
311
312         self.prepare_method(method)
313         self.prepare_url(url, params)
314         self.prepare_headers(headers)
315         self.prepare_cookies(cookies)
316         self.prepare_body(data, files, json)
317         self.prepare_auth(auth, url) ★endpoint2-30
318
319         # Note that prepare_auth must be last to enable authentication schemes
320         # such as OAuth to work on a fully prepared request.
321
322         # This MUST go after prepare_auth. Authenticators could add a hook
323         self.prepare_hooks(hooks)
…
534     def prepare_auth(self, auth, url=''): ★endpoint2-31
535         """Prepares the given HTTP auth data."""
536
537         # If no Auth is explicitly provided, extract it from the URL first.
538         if auth is None:
539             url_auth = get_auth_from_url(self.url)
540             auth = url_auth if any(url_auth) else None
541
542         if auth:
543             if isinstance(auth, tuple) and len(auth) == 2:
544                 # special-case basic HTTP auth
545                 auth = HTTPBasicAuth(*auth) ★endpoint2-32
546
547             # Allow auth to make its changes.
548             r = auth(self) ★endpoint2-35
549
550             # Update self to reflect the auth changes.
551             self.__dict__.update(r.__dict__)
552
553             # Recompute Content-Length
554             self.prepare_content_length(self.body)

lib/python3.6/site-packages/requests/auth.py

 79 class HTTPBasicAuth(AuthBase):  ★endpoint2-33
 80     """Attaches HTTP Basic Authentication to the given Request object."""
 81
 82     def __init__(self, username, password): ★endpoint2-34
 83         self.username = username
 84         self.password = password
 85
 86     def __eq__(self, other):
 87         return all([
 88             self.username == getattr(other, 'username', None),
 89             self.password == getattr(other, 'password', None)
 90         ])
 91
 92     def __ne__(self, other):
 93         return not self == other
 94
 95     def __call__(self, r): ★endpoint2-36
 96         r.headers['Authorization'] = _basic_auth_str(self.username, self.password) ★endpoint2-37: ここでHeaderを書き換え(Basic <username+passwordをbase64エンコードした文字列>)
 97         return r

ここまで目を通して、ようやく.netrcファイルにたどり着いた。

~/.netrc

 $ cat ~/.netrc
 machine api.github.com
	login <ログインユーザ名>
	password <MY_PRIVATE_ACCESS_TOKEN>

バッチリ設定していた。

おわりに

シェルのrcファイルはじめ、ホームディレクトリの.ファイルを見直していこうと思う。