litellm/docs/extras/integrations/document_loaders/git.ipynb

Git

Git is a distributed version control system that tracks changes in any set of computer files, usually used for coordinating work among programmers collaboratively developing source code during software development.

This notebook shows how to load text files from Git repository.

Load existing repository from disk

In [ ]:

!pip install GitPython

In [2]:

from git import Repo

repo = Repo.clone_from(
    "https://github.com/hwchase17/langchain", to_path="./example_data/test_repo1"
)
branch = repo.head.reference

In [ ]:

from langchain.document_loaders import GitLoader

In [6]:

loader = GitLoader(repo_path="./example_data/test_repo1/", branch=branch)

In [8]:

data = loader.load()

In [ ]:

len(data)

In [9]:

print(data[0])

page_content='.venv\n.github\n.git\n.mypy_cache\n.pytest_cache\nDockerfile' metadata={'file_path': '.dockerignore', 'file_name': '.dockerignore', 'file_type': ''}

Clone repository from url

In [5]:

from langchain.document_loaders import GitLoader

In [2]:

loader = GitLoader(
    clone_url="https://github.com/hwchase17/langchain",
    repo_path="./example_data/test_repo2/",
    branch="master",
)

In [10]:

data = loader.load()

In [11]:

len(data)

Out [11]:

1074

Filtering files to load

In [12]:

from langchain.document_loaders import GitLoader

# eg. loading only python files
loader = GitLoader(
    repo_path="./example_data/test_repo1/",
    file_filter=lambda file_path: file_path.endswith(".py"),
)

In [ ]: