How to Extract Metadata from Git Repositories

Every git commit carries a structured set of fields beyond the code diff itself. An active repository with 500 or more commits per year builds up a rich dataset you can query, filter, and export.

Here's what you can extract from each commit:

• Commit hash (%H for full SHA-1, %h for abbreviated). The unique identifier for every snapshot in the repo's history.
• Author name and email (%an, %ae). The person who wrote the change. This is distinct from the committer, which matters for rebased or cherry-picked commits.
• Committer name and email (%cn, %ce). The person who applied the change. In most workflows these match the author, but they diverge during rebases, patches, and integrations.
• Author date and committer date (%ad, %cd). Two separate timestamps per commit. Author date records when the change was originally written. Committer date records when it was applied to the current branch.
• Subject and body (%s, %b). The first line of the commit message and everything after the blank line.
• Diff stats (--stat, --shortstat, --numstat). Lines added, lines removed, and files changed per commit. This is where you get velocity and churn data.
• Branch and tag refs (%D). Which branches and tags point at the commit.
• GPG signature status (%G?, %GS, %GK). Whether the commit is signed, who signed it, and with which key.
• Tree hash (%T). The hash of the directory tree at that commit, useful for detecting identical snapshots across branches.
• Parent hashes (%P). References to parent commits, which reveal merge topology.

That's 15+ extractable fields per commit before you factor in file-level change data. The --numstat flag alone gives you per-file additions and deletions for every commit in the history.

Helpful references: Fast.io Workspaces, Fast.io Collaboration, and Fast.io AI.

The git log --pretty=format command is the most direct way to pull structured data from a repository. You define a template using placeholder tokens, and git outputs one line per commit in your chosen format.

Exporting to CSV

A basic CSV export captures the essentials:

git log --pretty=format:'%H,%an,%ae,%ad,%s' --date=iso > commits.csv

This gives you the full hash, author name, email, ISO-formatted date, and subject line for every commit. Add --numstat to append per-file change counts, though the output format becomes multi-line and needs post-processing.

For a more complete CSV with proper escaping, pipe through a formatter:

git log --pretty=format:'"%H","%an","%ae","%ad","%s"' \
  --date=iso-strict > commits.csv

The --date=iso-strict flag outputs RFC 3339 timestamps that parse cleanly in spreadsheets and databases.

Exporting to JSON

Git doesn't output native JSON, but the format string gets you close. Simon Willison documented a clean approach using jq:

git log --pretty=format:'%H%x00%an%x00%ae%x00%aI%x00%s' |
  jq -R -s 'split("
") | map(select(length > 0)) |
  map(split("")) |
  map({"hash": .[0], "author": .[1], "email": .[2],
       "date": .[3], "subject": .[4]})'

The %x00 placeholder inserts a null byte as the field separator, which avoids collisions with commas or quotes in commit messages. The jq command then splits on those null bytes and builds proper JSON objects.

Filtering by Date, Author, and Path

Git log accepts filters that narrow the extraction window:

git log --since="2025-01-01" --until="2026-01-01" \
  --author="alice@example.com" \
  --pretty=format:'%H,%an,%aI,%s' -- src/

This extracts only commits from 2025, by a specific author, touching files under src/. Combine --no-merges to skip merge commits if you want only direct contributions.

Adding Diff Stats

The --shortstat flag appends insertion and deletion counts:

git log --pretty=format:'COMMIT:%H,%an,%aI' --shortstat

Each commit prints on one line, followed by a stats line like 3 files changed, 47 insertions(+), 12 deletions(-). You'll need a script to merge these pairs into single records, but the data is there.

For anything beyond one-off exports, Python libraries make git metadata extraction repeatable and composable.

PyDriller

PyDriller is the standard library for mining software repositories in Python. It wraps GitPython with a higher-level API designed specifically for extraction and analysis.

from pydriller import Repository

for commit in Repository("/path/to/repo").traverse_commits():
    print(commit.hash, commit.author.name, commit.author_date)
    print(f"  Files: {len(commit.modified_files)}")
    for mod in commit.modified_files:
        print(f"  {mod.filename}: +{mod.added_lines} -{mod.deleted_lines}")

PyDriller gives you typed access to commit metadata, modified files, diffs, and even complexity metrics. It supports the same date, author, and path filters as git log, plus branch filtering and commit-range slicing.

Common extraction patterns with PyDriller:

• Contributor analysis: Group commits by author, count contributions, track active periods.
• Hotspot detection: Find files with the most modifications over a time window. High-churn files often correlate with bug density.
• Change coupling: Identify files that change together frequently, which reveals hidden dependencies.
• Commit message analysis: Parse conventional commit prefixes (feat, fix, chore) to categorize changes automatically.

GitPython GitPython provides lower-level access if you need to walk the object graph directly:

import git

repo = git.Repo("/path/to/repo")
for commit in repo.iter_commits("main", max_count=100):
    print(commit.hexsha, commit.author.name,
          commit.committed_datetime, commit.stats.total)

The commit.stats.total dictionary contains insertions, deletions, files changed, and lines per file. GitPython is more verbose than PyDriller but gives you direct access to trees, blobs, and refs when you need it.

Gimie

For standardized metadata extraction,

Gimie (Git Meta Information Extractor) pulls repository-level metadata and outputs it in the CodeMeta schema. It extracts licensing, contributor lists, language breakdown, and CI configuration from both the git index and the hosting platform (GitHub or GitLab). This is useful when you need to catalog repositories across an organization rather than analyze individual commits.

Store and Share Your Repository Analysis Reports

Upload extracted git metadata, compliance reports, and analytics dashboards to a shared workspace. Fast.io indexes everything automatically so your team can search and query reports without SQL access. 50 GB free, no credit card required.

Sign Up Free

Raw commit data becomes useful when you aggregate it into patterns. Here are the analyses that teams run most often on extracted git metadata.

Contribution Distribution

Export commits per author over time to see who is actively contributing and how workload distributes across the team:

git shortlog -sn --no-merges --since="2025-01-01"

This one-liner counts commits per author, sorted by volume. Pipe the output to a CSV for dashboard ingestion. For a time-series view, extract author and date fields and group by week or month in your analytics tool.

File Change Frequency

Files that change often are maintenance hotspots. Extract a ranked list:

git log --pretty=format: --name-only --since="2025-01-01" |
  sort | uniq -c | sort -rn | head -20

The top results often reveal configuration files, generated code, or modules that need refactoring. Cross-reference with --shortstat data to distinguish files that change frequently from files where changes are large.

Commit Velocity and Cadence

Plot commits per day or week to visualize development pace. Sudden drops can signal blockers. Spikes before deadlines flag potential quality risks. Extract the data with:

git log --pretty=format:'%aI' --since="2025-01-01" |
  cut -d'T' -f1 | sort | uniq -c

This gives you a date histogram: commit count per day, ready for charting.

Merge and Branch Topology

The --merges flag isolates merge commits, showing how branches flow into each other:

git log --merges --pretty=format:'%H,%P,%an,%aI,%s' > merges.csv

The parent hashes (%P) in merge commits reveal which branches were integrated and when. This data feeds compliance reports that need to prove code review happened before merges.

Extracting data is step one. The real value comes from routing it somewhere useful.

Databases and Data Warehouses

For ongoing analysis, load extracted commit data into PostgreSQL, BigQuery, or a similar store. A weekly cron job that runs your extraction script and appends new commits keeps the dataset current. Schema design is straightforward: a commits table with hash, author, date, subject, insertions, and deletions covers most queries.

PyDriller integrates well with pandas. Extract to a DataFrame and write directly to your database:

import pandas as pd
from pydriller import Repository

rows = []
for c in Repository("/path/to/repo").traverse_commits():
    rows.append({
        "hash": c.hash,
        "author": c.author.name,
        "date": c.author_date,
        "files_changed": len(c.modified_files),
        "insertions": c.insertions,
        "deletions": c.deletions
    })

df = pd.DataFrame(rows)
df.to_sql("commits", engine, if_exists="append", index=False)

Dashboards and Reporting

Grafana, Metabase, and Superset all connect to SQL databases. Once your commit data is in a table, build dashboards that track contributor activity, code churn, and release velocity without touching git directly.

For lighter setups, export to CSV and upload to a shared workspace where stakeholders can access reports directly. Fast.io workspaces work well here because files are automatically indexed once Intelligence is enabled, making reports searchable and queryable through AI chat. Team members can ask questions about the data without needing SQL access.

Compliance and Audit Trails

Regulated industries need proof that code changes went through review. Extracted merge metadata, GPG signature status, and branch protection evidence can be compiled into audit packages. Store these reports alongside source artifacts in a workspace with granular permissions so auditors see only what they need.

Fast.io's Metadata Views add another layer for teams that generate compliance documents alongside their code artifacts. Upload extracted reports and let Metadata Views pull structured fields (dates, approver names, signature status) into a sortable, filterable grid without manual data entry.

A few things that trip people up when extracting git metadata at scale.

Encoding Issues in Commit Messages

Commit messages can contain any UTF-8 characters, including commas, quotes, and newlines. If you're building CSV exports, use null-byte separators (%x00) instead of commas in your format string, then convert to proper CSV in a post-processing step. This avoids broken rows from messages that contain your delimiter.

Author vs. Committer Confusion In a rebase workflow, the author date and committer date diverge. The author date is when the change was originally written. The committer date is when the rebase applied it. If you're measuring "when was this work done," use author date. If you're measuring "when did this land on the branch," use committer date. Getting this wrong skews velocity metrics.

Shallow Clones Break History

Extraction CI systems often use --depth=1 shallow clones for speed. These contain only the most recent commit and have no history to extract. If your extraction pipeline runs in CI, configure a full clone or a blobless clone (--filter=blob:none), which fetches all commit metadata without downloading every file blob.

git clone --filter=blob:none https://github.com/org/repo.git

Blobless clones download commits and trees immediately but fetch file contents on demand. You get the full commit graph for metadata extraction without the storage cost of a complete clone.

Large Repositories and Performance

Repositories with hundreds of thousands of commits can make git log slow. Use --since and --until to limit the time window. If you need the full history, run the extraction once and then incrementally append new commits using git log HEAD..origin/main after each fetch.

PyDriller's from_commit and to_commit parameters serve the same purpose for Python-based extraction.