はじめに
PyGithubを利用して複数ユーザのリソースにアクセスしようとしたら、必ず同じユーザのリソースにしかアクセスできず途方にくれた。
認証にはユーザ毎に生成可能なPrivateAccessTokenを利用していたが、何度設定し直しても状況が変わらなかったため、仕方なくソースを読むことにした。
読み込むのに時間がかかったので、作業を無駄にしないためにもメモしておく。
結論から言えば.netrcファイルがホームディレクトリに配置されており、それが最優先されていたことが原因だった。
以前PyGithubと同じようなRubyのOctkitというツールを利用してGithubのAPIを利用した事があったが、おそらくそのタイミングで.netrcファイルを作ったのだろう。
そんなことをすっかり忘れ、しかもその設定が最優先されるなんて思いもよらなかった。当時の自分になにやってんだと言いたい。
実行環境
というわけで調査を開始したわけだが、その前にまず実行環境の確認から。
$ python --version
Python 3.6.5
$ pip list
Package Version
---------- --------
certifi 2019.3.9
chardet 3.0.4
Deprecated 1.2.5
idna 2.8
pip 19.0.3
PyGithub 1.43.5
PyJWT 1.7.1
requests 2.21.0
setuptools 39.0.1
urllib3 1.24.1
wrapt 1.11.1
調査
はじめに
今回調査に時間がかかってしまった理由は「実際にどんなリクエストが送信されているか」がすぐに分からなかったから。
こちらを参考にDEBUGレベルでログを出力しても、肝心のトークンはセキュリティを気にして表示してくれない。
当たり前のような機能だけど、今回に限ってはお節介も甚だしい。（完全に八つ当たりです）
実行手順
以下の要領で実行。
※enndpoint0-2は今後の説明のための実行の起点。
$ python
Python 3.6.5 (default, Dec 7 2018, 10:51:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>>
>>> from github import Github
>>>
>>> g = Github('<MY_PRIVATE_ACCESS_TOKEN>') #★endpoint0
>>> u = g.get_user() #★endpoint1
>>> u.login #★endpoint2
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443
DEBUG:urllib3.connectionpool:https://api.github.com:443 "GET /user HTTP/1.1" 200 None
DEBUG:github.Requester:GET https://api.github.com/user {'Authorization': 'token (oauth token removed)', 'User-Agent': 'PyGithub/Python'} None ==> 200 {'date': 'Mon, 01 Apr 2019 12:32:28 GMT', 'content-type': 'application/json; charset=utf-8', 'transfer-encoding': 'chunked', 'server': 'GitHub.com', 'status': '200 OK', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '4999', 'x-ratelimit-reset': '1554125548', 'cache-control': 'private, max-age=60, s-maxage=60', 'vary': 'Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding', 'etag': 'W/"97ff3b6da70a11af51d10e69099f2a96"', 'last-modified': 'Mon, 01 Apr 2019 03:06:05 GMT', 'x-oauth-scopes': 'admin:repo_hook, repo', 'x-accepted-oauth-scopes': '', 'x-github-media-type': 'github.v3; format=json', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'content-encoding': 'gzip', 'x-github-request-id': 'E056:8730:37E5769:43FEE21:5CA204DC'} {"login":"<ログインユーザ名>","id":<ユーザID:整数値>,"node_id":"<ノードID>","avatar_url":"https://avatars3.githubusercontent.com/u/<ユーザID:整数値>?v=4","gravatar_id":"","url":"https://api.github.com/users/<ログインユーザ名>","html_url":"https://github.com/<ログインユーザ名>","followers_url":"https://api.github.com/users/<ログインユーザ名>/followers","following_url":"https://api.github.com/users/<ログインユーザ名>/following{/other_user}","gists_url":"https://api.github.com/users/<ログインユーザ名>/gists{/gist_id}","starred_url":"https://api.github.com/users/<ログインユーザ名>/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/<ログインユーザ名>/subscriptions","organizations_url":"https://api.github.com/users/<ログインユーザ名>/orgs","repos_url":"https://api.github.com/users/<ログインユーザ名>/repos","events_url":"https://api.github.com/users/<ログインユーザ名>/events{/privacy}","received_events_url":"https://api.github.com/users/<ログインユーザ名>/received_events","type":"User","site_admin":false,"name":null,"company": null,"blog": null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":0,"public_gists":0,"followers":3,"following":0,"created_at":"2013-03-05T15:00:18Z","updated_at":"2019-04-01T03:06:05Z"}
'<ログインユーザ名>'
<MY_PRIVATE_ACCESS_TOKEN> がリソースを操作したいユーザのPrivateAccessToken。
本来なら <ログインユーザ名> にはこのアクセストークンに対応するユーザ名が表示されるのだが、実際には別のユーザの情報が表示された。
しかたなく、モジュールにprint文を挟んでデバッグすることに。
以下モジュールのソースコードを読んだ流れを記載する。
ソースコード探訪
実行手順のendpoint0-2から始まり、枝番の順に目を通した。
そして、最終的にはnetrcの設定がある場合に、RequestHeaderが上書きされている事がわかった。
lib/python3.6/site-packages/github/MainClass.py
93 class Github(object): ★endpoint0-1
94 """
95 This is the main class you instantiate to access the Github API v3. Optional parameters allow different authentication methods.
96 """
97
98 def __init__(self, login_or_token=None, password=None, jwt=None, base_url=DEFAULT_BASE_URL, timeout=DEFAULT_TIMEOUT, client_id=None, client_secret=None, user_agent='PyGithub/Python', per_page=DEFAULT_PER_PAGE, api_preview=False, verify=True): ★endpoint0-2
99 """
100 :param login_or_token: string
101 :param password: string
102 :param base_url: string
103 :param timeout: integer
104 :param client_id: string
105 :param client_secret: string
106 :param user_agent: string
107 :param per_page: int
108 :param verify: boolean or string
109 """
110
111 assert login_or_token is None or isinstance(login_or_token, str), login_or_token
112 assert password is None or isinstance(password, str), password
113 assert jwt is None or isinstance(jwt, str), jwt
114 assert isinstance(base_url, str), base_url
115 assert isinstance(timeout, int), timeout
116 assert client_id is None or isinstance(client_id, str), client_id
117 assert client_secret is None or isinstance(client_secret, str), client_secret
118 assert user_agent is None or isinstance(user_agent, str), user_agent
119 assert isinstance(api_preview, (bool))
120 self.__requester = Requester(login_or_token, password, jwt, base_url, timeout, client_id, client_secret, user_agent, per_page, api_preview, verify) ★endpoint0-3
…
220 def get_user(self, login=github.GithubObject.NotSet): ★endpoint1-1
221 """
222 :calls: `GET /users/:user <http://developer.github.com/v3/users>`_ or `GET /user <http://developer.github.com/v3/users>`_
223 :param login: string
224 :rtype: :class:`github.NamedUser.NamedUser`
225 """
226 assert login is github.GithubObject.NotSet or isinstance(login, str), login
227 if login is github.GithubObject.NotSet:
228 return AuthenticatedUser.AuthenticatedUser(self.__requester, {}, {"url": "/user"}, completed=False) ★endpoint1-2
229 else:
230 headers, data = self.__requester.requestJsonAndCheck(
231 "GET",
232 "/users/" + login
233 )
234 return github.NamedUser.NamedUser(self.__requester, headers, data, completed=True)
lib/python3.6/site-packages/github/AuthenticatedUser.py
63 class AuthenticatedUser(github.GithubObject.CompletableGithubObject): ★endpoint1-3
64 """
65 This class represents AuthenticatedUsers as returned by https://developer.github.com/v3/users/#get-the-authenticated-user
66
67 An AuthenticatedUser object can be created by calling ``get_user()`` on a Github object.
68 """
69
70 def __repr__(self):
71 return self.get__repr__({"login": self._login.value})
…
225 @property
226 def login(self): ★endpoint2-1
227 """
228 :type: string
229 """
230 self._completeIfNotSet(self._login) ★endpoint2-2
231 return self._login.value
lib/python3.6/site-packages/github/GithubObject.py
69 class GithubObject(object): ★endpoint1-6
70 """
71 Base class for all classes representing objects returned by the API.
72 """
73
74 '''
75 A global debug flag to enable header validation by requester for all objects
76 '''
77 CHECK_AFTER_INIT_FLAG = False
78
79 @classmethod
80 def setCheckAfterInitFlag(cls, flag):
81 cls.CHECK_AFTER_INIT_FLAG = flag
82
83 def __init__(self, requester, headers, attributes, completed): ★endpoint1-7
84 self._requester = requester
85 self._initAttributes()
86 self._storeAndUseAttributes(headers, attributes)
87
88 # Ask requester to do some checking, for debug and test purpose
89 # Since it's most handy to access and kinda all-knowing
90 if self.CHECK_AFTER_INIT_FLAG: # pragma no branch (Flag always set in tests)
91 requester.check_me(self)
…
250 class CompletableGithubObject(GithubObject): ★endpoint1-4
251 def __init__(self, requester, headers, attributes, completed):
252 GithubObject.__init__(self, requester, headers, attributes, completed) ★endpoint1-5
253 self.__completed = completed
…
261 def _completeIfNotSet(self, value): ★endpoint2-3
262 if value is NotSet:
263 self._completeIfNeeded() ★endpoint2-4
264
265 def _completeIfNeeded(self): ★endpoint2-5
266 if not self.__completed:
267 self.__complete() ★endpoint2-6
268
269 def __complete(self): ★endpoint2-7
270 headers, data = self._requester.requestJsonAndCheck(
271 "GET",
272 self._url.value
273 ) ★endpoint2-8
274 self._storeAndUseAttributes(headers, data)
275 self.__completed = True
lib/python3.6/site-packages/github/Requester.py
90 class HTTPSRequestsConnectionClass(object):
91 # mimic the httplib connection object
92 def __init__(self, host, port=None, strict=False, timeout=None, **kwargs):
93 self.port = port if port else 443
94 self.host = host
95 self.protocol = "https"
96 self.timeout = timeout
97 self.verify = kwargs.get("verify", True)
98 self.session = requests.Session()
99
100 def request(self, verb, url, input, headers):
101 self.verb = verb
102 self.url = url
103 self.input = input
104 self.headers = headers
105
106 def getresponse(self): ★endpoint2-19
107 verb = getattr(self.session, self.verb.lower()) ★ ★endpoint2-20: Sessionクラスのgetメソッドを取得
108 url = "%s://%s:%s%s" % (self.protocol, self.host, self.port, self.url)
109 r = verb(url, headers=self.headers, data=self.input, timeout=self.timeout, verify=self.verify, allow_redirects=False) ★endpoint2-21: getメソッド実行
110 return RequestsResponse(r)
111
112 def close(self):
113 return
…
142 class Requester: ★endpoint0-4
143 __httpConnectionClass = HTTPRequestsConnectionClass
144 __httpsConnectionClass = HTTPSRequestsConnectionClass ★endpoint0-6(last)
…
217 def __init__(self, login_or_token, password, jwt, base_url, timeout, client_id, client_secret, user_agent, per_page, api_preview, verify):
218 self._initializeDebugFeature()
219
220 if password is not None:
221 login = login_or_token
222 if atLeastPython3:
223 self.__authorizationHeader = "Basic " + base64.b64encode((login + ":" + password).encode("utf-8")).decode("utf-8").replace('\n', '') # pragma no cover (Covered by Authentication.testAuthorizationHeaderWithXxx with Python 3)
224 else:
225 self.__authorizationHeader = "Basic " + base64.b64encode(login + ":" + password).replace('\n', '')
226 elif login_or_token is not None:
227 token = login_or_token
228 self.__authorizationHeader = "token " + token
229 elif jwt is not None:
230 self.__authorizationHeader = "Bearer " + jwt
231 else:
232 self.__authorizationHeader = None
233
234 self.__base_url = base_url
235 o = urllib.parse.urlparse(base_url)
236 self.__hostname = o.hostname
237 self.__port = o.port
238 self.__prefix = o.path
239 self.__timeout = timeout
240 self.__scheme = o.scheme
241 if o.scheme == "https":
242 self.__connectionClass = self.__httpsConnectionClass ★endpoint0-5
243 elif o.scheme == "http":
244 self.__connectionClass = self.__httpConnectionClass
245 else:
246 assert False, "Unknown URL scheme"
…
263 def requestJsonAndCheck(self, verb, url, parameters=None, headers=None, input=None): ★endpoint2-9
264 return self.__check(*self.requestJson(verb, url, parameters, headers, input, self.__customConnection(url))) ★endpoint2-10
…
320 def requestJson(self, verb, url, parameters=None, headers=None, input=None, cnx=None): ★endpoint2-11
321 def encode(input):
322 return "application/json", json.dumps(input)
323
324 return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode) ★endpoint2-12
…
356 def __requestEncode(self, cnx, verb, url, parameters, requestHeaders, input, encode): ★endpoint2-13
357 assert verb in ["HEAD", "GET", "POST", "PATCH", "PUT", "DELETE"]
…
376
377 status, responseHeaders, output = self.__requestRaw(cnx, verb, url, requestHeaders, encoded_input) ★endpoint2-14
…
391 def __requestRaw(self, cnx, verb, url, requestHeaders, input): ★endpoint2-15
392 original_cnx = cnx
393 if cnx is None:
394 cnx = self.__createConnection() ★endpoint2-16
395 cnx.request(
396 verb,
397 url,
398 input,
399 requestHeaders
400 )
401 response = cnx.getresponse() ★endpoint2-20
…
452 def __createConnection(self): ★endpoint2-17
453 kwds = {}
454 if not atLeastPython3: # pragma no branch (Branch useful only with Python 3)
455 kwds["strict"] = True # Useless in Python3, would generate a deprecation warning
456 kwds["timeout"] = self.__timeout
457 kwds["verify"] = self.__verify
458
459 if self.__persist and self.__connection is not None:
460 return self.__connection
461
462 self.__connection = self.__connectionClass(self.__hostname, self.__port, **kwds) ★endpoint2-18(endpoint0-5で設定したもの)
463
464 return self.__connection ★endpoint2-19
lib/python3.6/site-packages/requests/sessions.py
340 class Session(SessionRedirectMixin):
…
426 def prepare_request(self, request): ★endpoint2-26
427 """Constructs a :class:`PreparedRequest <PreparedRequest>` for
428 transmission and returns it. The :class:`PreparedRequest` has settings
429 merged from the :class:`Request <Request>` instance and those of the
430 :class:`Session`.
431
432 :param request: :class:`Request` instance to prepare with this
433 session's settings.
434 :rtype: requests.PreparedRequest
435 """
436 cookies = request.cookies or {}
437
438 # Bootstrap CookieJar.
439 if not isinstance(cookies, cookielib.CookieJar):
440 cookies = cookiejar_from_dict(cookies)
441
442 # Merge with session cookies
443 merged_cookies = merge_cookies(
444 merge_cookies(RequestsCookieJar(), self.cookies), cookies)
445
446 # Set environment's basic authentication if not explicitly set.
447 auth = request.auth
448 if self.trust_env and not auth and not self.auth:
449 auth = get_netrc_auth(request.url) ★endpoint2-27: ここでnetrcの設定を取得
450
451 p = PreparedRequest()
452 p.prepare(
453 method=request.method.upper(),
454 url=request.url,
455 files=request.files,
456 data=request.data,
457 json=request.json,
458 headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
459 params=merge_setting(request.params, self.params),
460 auth=merge_setting(auth, self.auth),
461 cookies=merged_cookies,
462 hooks=merge_hooks(request.hooks, self.hooks),
463 ) ★endpoint2-28
464 return p
465
466 def request(self, method, url,
467 params=None, data=None, headers=None, cookies=None, files=None,
468 auth=None, timeout=None, allow_redirects=True, proxies=None,
469 hooks=None, stream=None, verify=None, cert=None, json=None): ★endpoint2-24
…
506 # Create the Request.
507 req = Request(
508 method=method.upper(),
509 url=url,
510 headers=headers,
511 files=files,
512 data=data or {},
513 json=json,
514 params=params or {},
515 auth=auth,
516 cookies=cookies,
517 hooks=hooks,
518 )
519 prep = self.prepare_request(req) ★endpoint2-25
…
537 def get(self, url, **kwargs): ★endpoint2-22
538 r"""Sends a GET request. Returns :class:`Response` object.
539
540 :param url: URL for the new :class:`Request` object.
541 :param \*\*kwargs: Optional arguments that ``request`` takes.
542 :rtype: requests.Response
543 """
544
545 kwargs.setdefault('allow_redirects', True)
546 return self.request('GET', url, **kwargs) ★endpoint2-23
lib/python3.6/site-packages/requests/models.py
272 class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
…
307 def prepare(self,
308 method=None, url=None, headers=None, files=None, data=None,
309 params=None, auth=None, cookies=None, hooks=None, json=None): ★endpoint2-29
310 """Prepares the entire request with the given parameters."""
311
312 self.prepare_method(method)
313 self.prepare_url(url, params)
314 self.prepare_headers(headers)
315 self.prepare_cookies(cookies)
316 self.prepare_body(data, files, json)
317 self.prepare_auth(auth, url) ★endpoint2-30
318
319 # Note that prepare_auth must be last to enable authentication schemes
320 # such as OAuth to work on a fully prepared request.
321
322 # This MUST go after prepare_auth. Authenticators could add a hook
323 self.prepare_hooks(hooks)
…
534 def prepare_auth(self, auth, url=''): ★endpoint2-31
535 """Prepares the given HTTP auth data."""
536
537 # If no Auth is explicitly provided, extract it from the URL first.
538 if auth is None:
539 url_auth = get_auth_from_url(self.url)
540 auth = url_auth if any(url_auth) else None
541
542 if auth:
543 if isinstance(auth, tuple) and len(auth) == 2:
544 # special-case basic HTTP auth
545 auth = HTTPBasicAuth(*auth) ★endpoint2-32
546
547 # Allow auth to make its changes.
548 r = auth(self) ★endpoint2-35
549
550 # Update self to reflect the auth changes.
551 self.__dict__.update(r.__dict__)
552
553 # Recompute Content-Length
554 self.prepare_content_length(self.body)
lib/python3.6/site-packages/requests/auth.py
79 class HTTPBasicAuth(AuthBase): ★endpoint2-33
80 """Attaches HTTP Basic Authentication to the given Request object."""
81
82 def __init__(self, username, password): ★endpoint2-34
83 self.username = username
84 self.password = password
85
86 def __eq__(self, other):
87 return all([
88 self.username == getattr(other, 'username', None),
89 self.password == getattr(other, 'password', None)
90 ])
91
92 def __ne__(self, other):
93 return not self == other
94
95 def __call__(self, r): ★endpoint2-36
96 r.headers['Authorization'] = _basic_auth_str(self.username, self.password) ★endpoint2-37: ここでHeaderを書き換え(Basic <username+passwordをbase64エンコードした文字列>)
97 return r
ここまで目を通して、ようやく.netrcファイルにたどり着いた。
~/.netrc
$ cat ~/.netrc
machine api.github.com
login <ログインユーザ名>
password <MY_PRIVATE_ACCESS_TOKEN>
バッチリ設定していた。
おわりに
シェルのrcファイルはじめ、ホームディレクトリの.ファイルを見直していこうと思う。