Motivation

Some time ago when I asked one of my colleagues, what whas his main motivation for becoming a programmer. He then replied in a sense that he wants his code be part of a software which can be useful for many years.

Since my colleague worked as a Web Developer at that time, I was pretty skeptical of this answer. In my previous experience, the front-end part of an average project tends to be rewritten every three months or so, so this is not a very good place to look for "stable" and "unchanging" code.

Later, however, I became more curious and decided to check, how long on average the line of code lives in our company's repository. The additional benefit was that I got myself a great excuse to play around with GitPython package during my work time!

If you have not heard this name before, GitPython provides a Python-interface to git. Since we will use it heavily below, the basic familiarity is assumed and welcomed. I will also use pandas below pretty much as well.

Finally, one last word before we embark on a journey. While working on this, I made myself an explicit goal to NOT to use anything except GitPython and pandas (remember, one of my goals was to learn the former). However, if you look for something more user-friendly, there are other packages built on top of GitPython, which provide much more rich and friendly interface. In particular, PyDriller popped out during my searches. But perhaps, plenty of others exist too.

Main work

Ok, so here we go. First, we initialize Repo object which will represent our repository. Make sure you have downloaded the repository and checked out the newest version. Replace PATH_TO_REPO below with path to your git repository on your disk.

from git import Repo

PATH_TO_REPO = "/Users/nailbiter/Documents/datawise/dtws-rdemo"
repo = Repo("/Users/nailbiter/Documents/datawise/dtws-rdemo")
assert not repo.bare

Next, we check out the branch we want to investigate (see the variable BRANCH below). In your case it probably will be master, but our main branch is called development for some reasons.

import pandas as pd
BRANCH = "development"
head = repo.commit(f"origin/{BRANCH}")
from IPython.display import HTML

def head_to_record(head):
    return {"sha":head.hexsha[:7],
            "parent(s)":[sha.hexsha[:7] for sha in head.parents],
            "name":head.message.strip(),
            "commit_obj":head
           }

records = []
while head.parents:
#     print(f"parent(s) of {head.hexsha} is/are {[h.hexsha for h in head.parents]}")
    records.append(head_to_record(head))
    head = head.parents[0]
records.append(head_to_record(head))

pd.DataFrame(records)

	sha	parent(s)	name	commit_obj
0	31ad850	[c77bfb0]	docs	31ad850b08014bbf299e534e28cdfee32be90654
1	c77bfb0	[d4935dc]	stash	c77bfb02b6aa0992be7d51ddc09c295a9b25d4d1
2	d4935dc	[f294f04]	rename owner	d4935dc2157c6f968db8bae7d68868955c06f6ea
3	f294f04	[c51257b]	stash	f294f049161ac9c8c2215f33b8d0bc25f49f88b3
4	c51257b	[b684146]	stash	c51257b1b89ea10bd213ed5ba575033fd0514e89
...	...	...	...	...
298	e636c9f	[0f9ad8d]	[Task] Ran Prettier	e636c9ff7f9125064c2f3d367680107baebae250
299	0f9ad8d	[ec1c72f]	[Task] Setup Prettier	0f9ad8d85d18010098228cabae36e674b4402686
300	ec1c72f	[e4da3e5]	[Feat.] Can Set Initial State	ec1c72fe382a5b9dfa42a8810a740f96eb72c05c
301	e4da3e5	[b17eb26]	Initial DTWS commit	e4da3e5ae67322feae1b93e2180b219054339182
302	b17eb26	[]	Initial commit from Create React App	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82

303 rows × 4 columns

Now, variables records in the code above represents all the commits on development branch since the beginning till the current moment.

Next we need to go along these commits and collect info regarding every line which appeared/disappeared in that commit. This information later will help us to determine lifetime of each line which ever appeared in our repository.

To do so, we create the variable res which is a dictionary. It's keys are tuples of the form (<line_content>,<commit>,<filename>) and its values are sets containing hashes of all the commits in which this line appeared. It is a rather big structure and computing it takes some time.

Therefore, be ready that the code below will take some time to finish (around 100 seconds on my reasonably new MacBook Pro with our repository having only ~300 commits).

I guess, there should be much more effective and elegant way to collect this data, but I have not came up with it yet. Suggestions are welcomed.

import pandas as pd
from tqdm import tqdm

def collect_filestates(end,start=None):
    """this procedure collects names of all files which changed from commit `start`
    till commit `end` (these assumed to be adjacent)"""
    if start is not None:
        diffs = start.diff(other=end)
        fns = [diff.b_path for diff in diffs]
        change_types = [diff.change_type for diff in diffs]
        res = [{"filename":t[0],"status":t[1]} for t in zip(fns,change_types)]
        return res
    else: 
        fns = end.stats.files.keys()
        return [{"filename":f,"status":"C"} for f in fns]
    
def collect_lines(end,start=None):
    """collects information about all lines that changed from `start` to `end`"""
    filestates = [r for r in collect_filestates(end,start) if r["status"] != "D"]
    res = {}
    for fs in filestates:
        fn = fs["filename"]
        blame = repo.blame(end,file=fn)
        for k,v in blame:
            for vv in v:
                res[(vv,k.hexsha,fn)] = end.hexsha
    return res

res = {}
for i in tqdm(range(len(records))):
    _res = collect_lines(end=records[i]["commit_obj"],start=None if (i+1)==len(records) else records[i+1]["commit_obj"])
    for k,v in _res.items():
        if k in res:
            res[k].add(v)
        else:
            res[k] = {v}
            
{k:v for k,v in list(res.items())[:5]}

100%|██████████| 303/303 [01:41<00:00,  2.99it/s]





{('*.swo',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386',
  'pipeline/.gitignore'): {'31ad850b08014bbf299e534e28cdfee32be90654',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386'},
 ('*.swp',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386',
  'pipeline/.gitignore'): {'31ad850b08014bbf299e534e28cdfee32be90654',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386'},
 ('.pulled_data.json',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386',
  'pipeline/.gitignore'): {'31ad850b08014bbf299e534e28cdfee32be90654',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386'},
 ('.config.custom.json',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386',
  'pipeline/.gitignore'): {'31ad850b08014bbf299e534e28cdfee32be90654',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386'},
 ('.stderr.txt',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386',
  'pipeline/.gitignore'): {'31ad850b08014bbf299e534e28cdfee32be90654',
  'a73ec421cfc05cc3816cb3b2b2505d228e60c386'}}

Now, as we have our marvelous res structure, we can easily compute the lifetime of every line which ever appeared in our repository: for every key in res we simply compute the duration between oldest and newest commit in its value set.

But again, this may take some time (around 6 minutes on my machine).

from datetime import datetime
import pandas as pd
from tqdm import tqdm

_records = []
for k in tqdm(res):
    dates = [datetime.fromtimestamp(repo.commit(sha).committed_date) for sha in res[k]]
    _records.append(dict(line=k[0],commit=k[1],file=k[2],lifetime=max(dates)-min(dates)))
lines_df = pd.DataFrame(_records)
lines_df

100%|██████████| 1806272/1806272 [05:52<00:00, 5123.11it/s]

	line	commit	file	lifetime
0	*.swo	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29
1	*.swp	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29
2	.pulled_data.json	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29
3	.config.custom.json	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29
4	.stderr.txt	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29
...	...	...	...	...
1806267	supports-color@^5.3.0:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00
1806268	through@^2.3.6:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00
1806269	typedarray@^0.0.6:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00
1806270	whatwg-fetch@3.0.0:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00
1806271	xtend@^4.0.0, xtend@~4.0.1:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00

1806272 rows × 4 columns

The table lines_df we assembled above contains the following columns:

line -- that's line's content
commit -- that's the first commit in which this line appeared.
file -- filename in which this line appears
lifetime -- lifetime of a line

In the code below we add two more columns to this table:

author -- author of the line (to protect their privacy, I do not list real names, but rather one-letter nicknames)
ext -- file extension of filename

from os.path import splitext, isfile
import json

if not isfile("author_masks.json"):
    to_author = lambda s:s
else:
    with open("author_masks.json") as f:
        d = json.load(f)
        to_author = lambda s:d[s]

lines_df.sort_values(by="lifetime",ascending=False)
lines_df["author"] = [to_author(str(repo.commit(sha).author)) for sha in lines_df["commit"]]
lines_df["ext"] = [splitext(fn)[1] for fn in lines_df["file"]]
lines_df

	line	commit	file	lifetime	author	ext
0	*.swo	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29	L
1	*.swp	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29	L
2	.pulled_data.json	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29	L
3	.config.custom.json	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29	L
4	.stderr.txt	a73ec421cfc05cc3816cb3b2b2505d228e60c386	pipeline/.gitignore	144 days 19:07:29	L
...	...	...	...	...	...	...
1806267	supports-color@^5.3.0:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00	J	.lock
1806268	through@^2.3.6:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00	J	.lock
1806269	typedarray@^0.0.6:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00	J	.lock
1806270	whatwg-fetch@3.0.0:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00	J	.lock
1806271	xtend@^4.0.0, xtend@~4.0.1:	b17eb263f8cf0adc7eb8cda7e0a0ae4aee8aff82	yarn.lock	0 days 00:00:00	J	.lock

1806272 rows × 6 columns

Analysis

Finally, having this info, we can then group, and average lifetime on various parameters.

For example, below, we see the average lifetime of every line conditional on file extension:

from datetime import timedelta
from functools import reduce

def averager(key, df=lines_df):
    ltk = "lifetime (days)"
    return pd.DataFrame([
        {key:ext,
         ltk:(reduce(lambda t1,t2:t1+t2.to_pytimedelta(),slc["lifetime"],timedelta())/len(slc)).days
        } 
        for ext,slc 
        in df.groupby(key)
    ]).set_index(key).sort_values(by=ltk,ascending=False)
averager("ext").plot.barh(figsize=(16,10),logx=True)

You can see that ironically, the lines that stay unchanged the longest, belong to "insignificant" files like .lock (that's various yarn.lock's), .rules (that's Firabase rules), .html (that's index.html and since our project uses React, the main index.html also receives almost no changes) and others. In particular, files with empty extension refer to .gitignore's.

And finally, we can see the average lifetime of a line of code, conditional on author.

averager("author").plot.barh(figsize=(16,10))

We can see that the colleague I mentioned in the beginning (he goes by the nickname "J" here) indeed authored the longest-surviving lines in the whole repository. Good for him.

However, let's look more closely at the secret of his success:

_df = lines_df[[ext=="J" for ext in lines_df["author"]]].loc[:,["line","file","lifetime","ext"]]
_df = averager(df=_df,key="ext")
_df[[x>0 for x in _df["lifetime (days)"]]]

	lifetime (days)
ext
.html	181
.lock	165
	154
.rules	91
.yml	54
.md	39
.ts	38
.js	26
.json	18
.snap	11
.css	5
.tsx	4

Being the founder of the repository under consideration, he in particular mostly authored the aforementioned index.html, yarn.lock and *.rules files. As I explained before, these received almost no changes during the subsequent development.

Further work

Since we store the info on filenames as well, we can compute the averages conditional on folders, thus seeing, which parts of project are more "stable" than the others.