Git Blame Cache

[holodorum]

Summary 💡

Would there be any interest for a git blame cache. I came across this thread, where this is discussed with some links to existing implementations.
This thesis also implemented something similar for a company.
With the recent improvements in speed achieved by @cruessler I think this might be a nice addition to beat git and I would be interested to give implementing it a shot.

Motivation 🔦

No response

[jtwaleson]

Adding a bit of context here, as I worked on this with @holodorum .

I'm working on a product that tracks code ownership / code age for all files in many repos. For that, we'd like to have very fast git blame statistics. The solution we came up with is to create checkpoints of specific commits and store the full blame coverage for those. When we want to calculate the blame for a newer commit, we only have to go as far back as the latest checkpoint. It stores chunks with (commit_id, start_line, end_line) tuples.

We're storing the chunks in RocksDB so I think it's too specific to include in gitoxide itself, but we were wondering if there is some appetite to restructure the gix-blame code to make it easy to add a cache.

[Byron]

Thanks a lot for sharing - I am very interested in this line of work!
Right now, without a cache and from my imperfect memory, we are easily slower by a factor of 2x. This is due to doing more work than Git, usually, and by not being able to use certain optimizations. But even in plain blames without the use of the commit-graph, we are typically significantly slower than Git.

Making the code more cache-friendly and generally having more eyes concerned with performance on it is very much in my interest, so please feel free to join the efforts in making gix-blame as fast (and faster) than Git :D.

As for the implementation of a cache, there already is ein t query that builds up a sqlite database with a lot of expensive-to-compute derived data, some of which is also useful for blame. In theory, gix-blame could be abstracted more to allow using such caches, and possible also to more easily generate checkpoints.

Please note that making such changes should serve an actual product or tool so they serve a purpose/satisfy a demand, while having stakeholders that are interested in keeping it functioning.

[holodorum]

Thanks for your response! I've submitted an initial pull request that could speed up the blame process by using an existing blame as a starting point.

In the full implementation, we perform a tree diff between the commit for which we have a cache and the new commit. The files that were modified or added can then be blamed using the updated blame::function::file().

I wasn't aware of ein t query, it looks quite powerful. In our implementation, we utilized resource_cache to accelerate diffing. If I understand correctly, the resource cache optimizes the diffing of blobs, while ein t query is only beneficial to speed up tree diffs.

[jtwaleson]

Please note that making such changes should serve an actual product or tool so they serve a purpose/satisfy a demand, while having stakeholders that are interested in keeping it functioning.

@Byron It's definitely serving an actual problem for my product (in stealth), but it's a tiny bootstrapped startup so I can't guarantee any long-term commitment yet :) Please be aware of that when reviewing / accepting changes from our side.

[Byron]

Thanks for being so upfront about it, much appreciated!

Accepting cache-support in some shape or form should, and this is my hope, also mean that you can submit fixes and improvements to the core-algorithm in future, without having to fork and maintain your own.
If this resonates in some way it should work out fine.