digest.txt

Directory structure:
└── cyclotruc-gitingest/
    ├── CODE_OF_CONDUCT.md
    ├── pytest.ini
    ├── LICENSE
    ├── requirements.txt
    ├── Dockerfile
    ├── docs/
    ├── README.md
    ├── setup.py
    ├── SECURITY.md
    └── src/
        ├── routers/
        │   ├── index.py
        │   ├── download.py
        │   ├── __init__.py
        │   └── dynamic.py
        ├── config.py
        ├── __init__.py
        ├── process_query.py
        ├── server_utils.py
        ├── static/
        │   ├── js/
        │   │   ├── snow.js
        │   │   └── utils.js
        │   └── robots.txt
        ├── templates/
        │   ├── api.jinja
        │   ├── base.jinja
        │   ├── github.jinja
        │   ├── components/
        │   │   ├── github_form.jinja
        │   │   ├── navbar.jinja
        │   │   ├── result.jinja
        │   │   └── footer.jinja
        │   └── index.jinja
        ├── main.py
        └── gitingest/
            ├── ingest.py
            ├── ingest_from_query.py
            ├── tests/
            │   ├── conftest.py
            │   ├── __init__.py
            │   ├── test_clone.py
            │   ├── test_ingest.py
            │   └── test_parse_query.py
            ├── __init__.py
            ├── cli.py
            ├── utils.py
            ├── parse_query.py
            └── clone.py

================================================
File: /CODE_OF_CONDUCT.md
================================================
# Contributor Covenant Code of Conduct

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

## Our Standards

Examples of behavior that contributes to a positive environment for our
community include:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
  and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
  overall community

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or
  advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
  address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
  professional setting

## Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.

Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.

## Scope

This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
romain@coderamp.io.
All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the
reporter of any incident.

## Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:

### 1. Correction

**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.

**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.

### 2. Warning

**Community Impact**: A violation through a single incident or series
of actions.

**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.

### 3. Temporary Ban

**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.

**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.

### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior,  harassment of an
individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within
the community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.

Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).

[homepage]: https://www.contributor-covenant.org

For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.


================================================
File: /pytest.ini
================================================
[pytest]
pythonpath = src
testpaths = src/gitingest/tests
asyncio_mode = auto

# Coverage configuration
addopts = --no-cov

python_files = test_*.py
python_classes = Test*
python_functions = test_*

================================================
File: /LICENSE
================================================
MIT License

Copyright (c) 2024 Romain Courtois

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


================================================
File: /requirements.txt
================================================
fastapi[standard]
uvicorn
fastapi-analytics
slowapi
tiktoken
pytest
pytest-asyncio
click>=8.0.0


================================================
File: /Dockerfile
================================================
FROM python:3.12

WORKDIR /app

# Create a non-root user
RUN useradd -m -u 1000 appuser

COPY src/ ./
COPY requirements.txt ./

RUN pip install -r requirements.txt

# Change ownership of the application files
RUN chown -R appuser:appuser /app

# Switch to non-root user
USER appuser

EXPOSE 8000

CMD ["uvicorn", "main:app", "--reload", "--host", "0.0.0.0"]


================================================
File: /README.md
================================================
[![Image](./docs/frontpage.png "GitIngest main page")](https://gitingest.com/)

![License](https://img.shields.io/badge/license-MIT-blue.svg)

# GitIngest 🔍
Turn any Git repository into a prompt-friendly text ingest for LLMs.

You can also replace `hub` with `ingest` in any github url to access the coresponding digest

[gitingest.com](https://gitingest.com/) 


## 🚀 Features

- **Easy code context**: Get a text digest from a git repository URL or a directory
- **Smart Formatting**: Optimized output format for LLM prompts
- **Statistics about**: :
  - File and directory structure
  - Size of the extract
  - Token count  
- **CLI tool**: Run it as a command (Currently on Linux only)
- **Python package**: Import it in your code


## 📦 Installation

```
pip install gitingest
```


## 💡 Command Line usage

The `gitingest` command line tool allows you to analyze codebases and create a text dump of their contents.

```bash
# Basic usage
gitingest /path/to/directory

# From url
gitingest https://github.com/cyclotruc/gitingest

# See more options
gitingest --help
```

This will write the digest in a text file (default `digest.txt`) in your current working directory.


## 🐛 Python package usage


```python
from gitingest import ingest

summary, tree, content = ingest("path/to/directory")

#or from URL
summary, tree, content = ingest("https://github.com/cyclotruc/gitingest")
```

By default, this won't write a file but can be enabled with the `output` argument


## 🛠️ Using
- Tailwind CSS - Frontend
- [FastAPI](https://github.com/fastapi/fastapi) - Backend framework
- [tiktoken](https://github.com/openai/tiktoken) - Token estimation
- [apianalytics.dev](https://www.apianalytics.dev/) - Simple Analytics


## 🌐 Self-host 
1. Build the image:
```
docker build -t gitingest .
```

2. Run the container:
```
docker run -d --name gitingest -p 8000:8000 gitingest
```
The application will be available at `http://localhost:8000`
Ensure environment variables are set before running the application or deploying it via Docker.

## ✔️ Contributing

Contributions are welcome! 

Gitingest aims to be friendly for first time contributors, with a simple python and html codebase. If you need any help while working with the code, reach out to us on [discord](https://discord.com/invite/zerRaGK9EC)

### Ways to contribute 

1. Provide your feedback and ideas on discord
2. Open an Issue on github to report a bug 
2. Create a Pull request
   - Fork the repository
   - Make your changes and test them locally
   - Open a pull request for review and feedback

### 🔧 Local dev

####  Environment Configuration
- **`ALLOWED_HOSTS`**: Specify allowed hostnames for the application. Default: `"gitingest.com,*.gitingest.com,gitdigest.dev,localhost"`.
You can configure the application using the following environment variables:

```bash
ALLOWED_HOSTS="gitingest.local,localhost"
```

#### Run locally
1. Clone the repository 
```bash
git clone https://github.com/cyclotruc/gitingest.git
cd gitingest
```

2. Install dependencies
```bash
pip install -r requirements.txt
```

3. Run the application:
```bash
cd src
uvicorn main:app --reload
```

The frontend will be available at `localhost:8000` 




================================================
File: /setup.py
================================================
from setuptools import setup, find_packages

setup(
    name="gitingest",
    version="0.1.2",
    packages=find_packages(where="src"),
    package_dir={"": "src"},
    include_package_data=True,
    install_requires=[
        "click>=8.0.0",
        "tiktoken",
    ],
    entry_points={
        "console_scripts": [
            "gitingest=gitingest.cli:main",
        ],
    },
    python_requires=">=3.6",
    author="Romain Courtois",
    author_email="romain@coderamp.io",
    description="CLI tool to analyze and create text dumps of codebases for LLMs",
    long_description=open("README.md").read(),
    long_description_content_type="text/markdown",
    url="https://github.com/cyclotruc/gitingest",
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
    ],
)

================================================
File: /SECURITY.md
================================================
# Security Policy

## Reporting a Vulnerability

If you have discovered a vulnerability inside the project, report it privately at romain@coderamp.io. This way the maintainer can work on a proper fix without disclosing the problem to the public before it has been solved.


================================================
File: /src/routers/index.py
================================================
from fastapi import APIRouter, Request, Form
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates

from server_utils import limiter
from process_query import process_query
from config import EXAMPLE_REPOS


router = APIRouter()
templates = Jinja2Templates(directory="templates")


@router.get("/", response_class=HTMLResponse)
async def home(request: Request):
    return templates.TemplateResponse(
        "index.jinja", 
        {
            "request": request,
            "examples": EXAMPLE_REPOS,
            "default_file_size": 243
        }
    )


@router.post("/", response_class=HTMLResponse)
@limiter.limit("10/minute") 
async def index_post(
    request: Request, 
    input_text: str = Form(...),
    max_file_size: int = Form(...),
    pattern_type: str = Form(...),
    pattern: str = Form(...)
):
    return await process_query(request, input_text, max_file_size, pattern_type, pattern, is_index=True)
    
    





================================================
File: /src/routers/download.py
================================================
from fastapi import HTTPException, APIRouter
from fastapi.responses import Response
from config import TMP_BASE_PATH
import os

router = APIRouter()

@router.get("/download/{digest_id}")
async def download_ingest(digest_id: str):
    try:
        # Find the first .txt file in the directory
        directory = f"{TMP_BASE_PATH}/{digest_id}"
        txt_files = [f for f in os.listdir(directory) if f.endswith('.txt')]
        
        if not txt_files:
            raise FileNotFoundError("No .txt file found")
            
        with open(f"{directory}/{txt_files[0]}", "r") as f:
            content = f.read()
        
        return Response(
            content=content,
            media_type="text/plain",
            headers={
                "Content-Disposition": f"attachment; filename={txt_files[0]}"
            }
        )
    except FileNotFoundError:
        raise HTTPException(status_code=404, detail="Digest not found")

================================================
File: /src/routers/__init__.py
================================================
from .download import router as download
from .dynamic import router as dynamic
from .index import router as index

__all__ = ["download", "dynamic", "index"]

================================================
File: /src/routers/dynamic.py
================================================
from fastapi import APIRouter, Request, Form
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates

from process_query import process_query
from server_utils import limiter

router = APIRouter()
templates = Jinja2Templates(directory="templates")

@router.get("/{full_path:path}")
async def catch_all(request: Request, full_path: str):
    return templates.TemplateResponse(
        "github.jinja",
        {
            "request": request,
            "github_url": f"https://github.com/{full_path}",
            "loading": True,
            "default_file_size": 243
        }
    )

@router.post("/{full_path:path}", response_class=HTMLResponse)
@limiter.limit("10/minute") 
async def process_catch_all(
    request: Request, 
    input_text: str = Form(...),
    max_file_size: int = Form(...),
    pattern_type: str = Form(...),
    pattern: str = Form(...)
):
    return await process_query(request, input_text, max_file_size, pattern_type, pattern, is_index=False)
    

================================================
File: /src/config.py
================================================
MAX_DISPLAY_SIZE = 300000
TMP_BASE_PATH = "../tmp"

EXAMPLE_REPOS = [
    {"name": "Gitingest", "url": "https://github.com/cyclotruc/gitingest"},
    {"name": "FastAPI", "url": "https://github.com/tiangolo/fastapi"},
    {"name": "Flask", "url": "https://github.com/pallets/flask"},
    {"name": "Tldraw", "url": "https://github.com/tldraw/tldraw"},
    {"name": "ApiAnalytics", "url": "https://github.com/tom-draper/api-analytics"},
]


================================================
File: /src/process_query.py
================================================
from typing import List
from fastapi.templating import Jinja2Templates
from fastapi import Request

from config import MAX_DISPLAY_SIZE, EXAMPLE_REPOS
from gitingest import ingest_from_query, clone_repo, parse_query
from server_utils import logSliderToSize

templates = Jinja2Templates(directory="templates")

async def process_query(request: Request, input_text: str, slider_position: int, pattern_type: str = "exclude", pattern: str = "", is_index: bool = False) -> str:
    template = "index.jinja" if is_index else "github.jinja"
    max_file_size = logSliderToSize(slider_position)
    if pattern_type == "include":
        include_patterns = pattern
        exclude_patterns = None
    elif pattern_type == "exclude":
        exclude_patterns = pattern
        include_patterns = None
    try:
        query = parse_query(input_text, max_file_size, True, include_patterns, exclude_patterns)
        await clone_repo(query)
        summary, tree, content = ingest_from_query(query)
        with open(f"{query['local_path']}.txt", "w") as f:
            f.write(tree + "\n" + content)
        print(f"{query['slug']:<20}", end="")
        if pattern and pattern != "":
            print(f"{pattern_type}[{pattern}]", end="")
        print(f"\n{query['url']}")
    except Exception as e:
        return templates.TemplateResponse(
            template, 
            {
                "request": request,
                "github_url": input_text,
                "error_message": f"Error: {e}",
                "examples": EXAMPLE_REPOS if is_index else [],
                "default_file_size": slider_position,
                "pattern_type": pattern_type,
                "pattern": pattern,
            }
        )
    
    if len(content) > MAX_DISPLAY_SIZE:
        content = f"(Files content cropped to {int(MAX_DISPLAY_SIZE/1000)}k characters, download full ingest to see more)\n" + content[:MAX_DISPLAY_SIZE]
        
    return templates.TemplateResponse(
        template, 
        {
            "request": request, 
            "github_url": input_text,
            "result": True, 
            "summary": summary,
            "tree": tree, 
            "content": content,
            "examples": EXAMPLE_REPOS if is_index else [],
            "ingest_id": query['id'],
            "default_file_size": slider_position,
            "pattern_type": pattern_type,
            "pattern": pattern,
        }
    )


================================================
File: /src/server_utils.py
================================================

## Rate Limiter
from slowapi import Limiter
from slowapi.util import get_remote_address
limiter = Limiter(key_func=get_remote_address)

## Logarithmic slider to file size
import math
def logSliderToSize(position):
    """Convert slider position to file size in KB"""
    maxp = 500
    minv = math.log(1)
    maxv = math.log(102400)
    
    return round(math.exp(minv + (maxv - minv) * pow(position / maxp, 1.5))) * 1024

================================================
File: /src/static/js/snow.js
================================================
// Snow effect initialization
function initSnow() {
    const snowCanvas = document.getElementById('snow-canvas');
    const ctx = snowCanvas.getContext('2d');

    // Configure snow
    const snowflakes = [];
    const maxSnowflakes = 50;
    const spawnInterval = 200;
    let currentSnowflakes = 0;
    let lastSpawnTime = 0;

    // Resize canvas to window size
    function resizeCanvas() {
        snowCanvas.width = window.innerWidth;
        snowCanvas.height = window.innerHeight;
    }

    // Initial setup
    resizeCanvas();
    window.addEventListener('resize', resizeCanvas);

    // Snowflake class definition
    class Snowflake {
        constructor() {
            this.reset();
        }

        reset() {
            this.x = Math.random() * snowCanvas.width;
            this.y = 0;
            this.size = Math.random() * 3 + 2;
            this.speed = Math.random() * 1 + 0.5;
            this.wind = Math.random() * 0.5 - 0.25;
        }

        update() {
            this.y += this.speed;
            this.x += this.wind;

            if (this.y > snowCanvas.height) {
                this.reset();
            }
        }

        draw() {
            ctx.save();

            ctx.shadowColor = 'rgba(0, 0, 0, 0.3)';
            ctx.shadowBlur = 5;
            ctx.shadowOffsetX = 2;
            ctx.shadowOffsetY = 2;

            ctx.beginPath();
            ctx.arc(this.x, this.y, this.size, 0, Math.PI * 2);
            ctx.fillStyle = 'rgba(255, 255, 255, 1)';
            ctx.fill();

            ctx.strokeStyle = 'rgba(200, 200, 200, 0.8)';
            ctx.lineWidth = 0.5;
            ctx.stroke();

            ctx.restore();
        }
    }

    function animate(currentTime) {
        ctx.clearRect(0, 0, snowCanvas.width, snowCanvas.height);

        if (currentSnowflakes < maxSnowflakes && currentTime - lastSpawnTime > spawnInterval) {
            snowflakes.push(new Snowflake());
            currentSnowflakes++;
            lastSpawnTime = currentTime;
        }

        snowflakes.forEach(snowflake => {
            snowflake.update();
            snowflake.draw();
        });

        requestAnimationFrame(animate);
    }

    requestAnimationFrame(animate);
}

// Initialize snow when DOM content is loaded
document.addEventListener('DOMContentLoaded', initSnow);

// Also initialize when the HTMX content is swapped
document.addEventListener('htmx:afterSettle', initSnow);

================================================
File: /src/static/js/utils.js
================================================
// Copy functionality
function copyText(className) {
    const textarea = document.querySelector('.' + className);
    const button = document.querySelector(`button[onclick="copyText('${className}')"]`);
    if (!textarea || !button) return;

    // Copy text
    navigator.clipboard.writeText(textarea.value)
        .then(() => {
            // Store original content
            const originalContent = button.innerHTML;

            // Change button content
            button.innerHTML = 'Copied!';

            // Reset after 1 second
            setTimeout(() => {
                button.innerHTML = originalContent;
            }, 1000);
        })
        .catch(err => {
            // Show error in button
            const originalContent = button.innerHTML;
            button.innerHTML = 'Failed to copy';
            setTimeout(() => {
                button.innerHTML = originalContent;
            }, 1000);
        });
}


function handleSubmit(event, showLoading = false) {
    event.preventDefault();
    const form = event.target || document.getElementById('ingestForm');
    if (!form) return;

    const submitButton = form.querySelector('button[type="submit"]');
    if (!submitButton) return;

    const formData = new FormData(form);

    // Update file size
    const slider = document.getElementById('file_size');
    if (slider) {
        formData.delete('max_file_size');
        formData.append('max_file_size', slider.value);
    }

    // Update pattern type and pattern
    const patternType = document.getElementById('pattern_type');
    const pattern = document.getElementById('pattern');
    if (patternType && pattern) {
        formData.delete('pattern_type');
        formData.delete('pattern');
        formData.append('pattern_type', patternType.value);
        formData.append('pattern', pattern.value);
    }

    const originalContent = submitButton.innerHTML;
    const currentStars = document.getElementById('github-stars')?.textContent;

    if (showLoading) {
        submitButton.disabled = true;
        submitButton.innerHTML = `
            <div class="flex items-center justify-center">
                <svg class="animate-spin h-5 w-5 text-gray-900" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24">
                    <circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"></circle>
                    <path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
                </svg>
                <span class="ml-2">Processing...</span>
            </div>
        `;
        submitButton.classList.add('bg-[#ffb14d]');
    }

    // Submit the form
    fetch(form.action, {
        method: 'POST',
        body: formData
    })
        .then(response => response.text())
        .then(html => {
            // Store the star count before updating the DOM
            const starCount = currentStars;


            // TEMPORARY SNOW LOGIC //
            const parser = new DOMParser();
            const newDoc = parser.parseFromString(html, 'text/html');

            const existingCanvas = document.getElementById('snow-canvas');
            document.body.innerHTML = newDoc.body.innerHTML;
            if (existingCanvas) {
                document.body.insertBefore(existingCanvas, document.body.firstChild);
            }
            // END TEMPORARY SNOW LOGIC //

            // Wait for next tick to ensure DOM is updated
            setTimeout(() => {
                // Reinitialize slider functionality
                initializeSlider();

                const starsElement = document.getElementById('github-stars');
                if (starsElement && starCount) {
                    starsElement.textContent = starCount;
                }

                // Scroll to results if they exist
                const resultsSection = document.querySelector('[data-results]');
                if (resultsSection) {
                    resultsSection.scrollIntoView({ behavior: 'smooth', block: 'start' });
                }
            }, 0);
        })
        .catch(error => {
            submitButton.disabled = false;
            submitButton.innerHTML = originalContent;
        });
}

function copyFullDigest() {
    const directoryStructure = document.querySelector('.directory-structure').value;
    const filesContent = document.querySelector('.result-text').value;
    const fullDigest = `${directoryStructure}\n\nFiles Content:\n\n${filesContent}`;
    const button = document.querySelector('[onclick="copyFullDigest()"]');
    const originalText = button.innerHTML;

    navigator.clipboard.writeText(fullDigest).then(() => {
        button.innerHTML = `
            <svg class="w-4 h-4 mr-2" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M5 13l4 4L19 7"></path>
            </svg>
            Copied!
        `;

        setTimeout(() => {
            button.innerHTML = originalText;
        }, 2000);
    }).catch(err => {
        console.error('Failed to copy text: ', err);
    });
}

// Add the logSliderToSize helper function
function logSliderToSize(position) {
    const minp = 0;
    const maxp = 500;
    const minv = Math.log(1);
    const maxv = Math.log(102400);

    const value = Math.exp(minv + (maxv - minv) * Math.pow(position / maxp, 1.5));
    return Math.round(value);
}

// Move slider initialization to a separate function
function initializeSlider() {
    const slider = document.getElementById('file_size');
    const sizeValue = document.getElementById('size_value');

    if (!slider || !sizeValue) return;

    function updateSlider() {
        const value = logSliderToSize(slider.value);
        sizeValue.textContent = formatSize(value);
        slider.style.backgroundSize = `${(slider.value / slider.max) * 100}% 100%`;
    }

    // Update on slider change
    slider.addEventListener('input', updateSlider);

    // Initialize slider position
    updateSlider();
}

// Add helper function for formatting size
function formatSize(sizeInKB) {
    if (sizeInKB >= 1024) {
        return Math.round(sizeInKB / 1024) + 'mb';
    }
    return Math.round(sizeInKB) + 'kb';
}

// Initialize slider on page load
document.addEventListener('DOMContentLoaded', initializeSlider);

// Make sure these are available globally
window.copyText = copyText;

window.handleSubmit = handleSubmit;
window.initializeSlider = initializeSlider;
window.formatSize = formatSize;

// Add this new function
function setupGlobalEnterHandler() {
    document.addEventListener('keydown', function (event) {
        if (event.key === 'Enter' && !event.target.matches('textarea')) {
            const form = document.getElementById('ingestForm');
            if (form) {
                handleSubmit(new Event('submit'), true);
            }
        }
    });
}

// Add to the DOMContentLoaded event listener
document.addEventListener('DOMContentLoaded', () => {
    initializeSlider();
    setupGlobalEnterHandler();
});


================================================
File: /src/static/robots.txt
================================================
User-agent: *
Allow: / 
Allow: /api/
Allow: /cyclotruc/gitingest/



================================================
File: /src/templates/api.jinja
================================================
{% extends "base.jinja" %}

{% block title %}Git ingest API{% endblock %}

{% block content %}
<div class="relative">
    <div class="w-full h-full absolute inset-0 bg-black rounded-xl translate-y-2 translate-x-2"></div>
    <div class="bg-[#fff4da] rounded-xl border-[3px] border-gray-900 p-8 relative z-20">
        <h1 class="text-3xl font-bold text-gray-900 mb-4">API Documentation</h1>


        <div class="prose prose-blue max-w-none">
            <div class="bg-yellow-50 border-[3px] border-gray-900 p-4 mb-6 rounded-lg">
                <div class="flex">
                    <div class="flex-shrink-0">
                        <svg class="h-5 w-5 text-yellow-400" viewBox="0 0 20 20" fill="currentColor">
                            <path fill-rule="evenodd"
                                d="M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z"
                                clip-rule="evenodd" />
                        </svg>
                    </div>
                    <div class="ml-3">
                        <p class="text-sm text-gray-900">
                            The API is currently under development..
                        </p>
                    </div>
                </div>
            </div>
            <p class="text-gray-900">
                We're working on making our API available to the public.
                In the meantime, you can
                <a href="https://github.com/cyclotruc/gitingest/issues/new" target="_blank"
                    rel="noopener noreferrer" class="text-[#6e5000] hover:underline">
                    open an issue on github
                </a>
                to suggest features.
            </p>
        </div>
    </div>
</div>
{% endblock %}

================================================
File: /src/templates/base.jinja
================================================
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="icon" type="image/x-icon" href="/static/favicon.ico">

    <!-- Search Engine Meta Tags -->
    <meta name="description" content="Replace 'hub' with 'ingest' in any Github Url for a prompt-friendly text">
    <meta name="keywords" content="GitIngest, AI tools, LLM integration, Ingest, Digest, Context, Prompt, Git workflow, codebase extraction, Git repository, Git automation, Summarize, prompt-friendly">
    <meta name="robots" content="index, follow">
    
    <!-- OpenGraph Meta Tags -->
    <meta property="og:title" content="{% block og_title %}Git ingest{% endblock %}">
    <meta property="og:description" content="{% block og_description %}Replace 'hub' with 'ingest' in any Github Url for a prompt-friendly text{% endblock %}">
    <meta property="og:type" content="{% block og_type %}website{% endblock %}">
    <meta property="og:url" content="{{ request.url }}">
    <meta property="og:image" content="{% block og_image %}/static/og-image.png{% endblock %}">
    
    <title>{% block title %}Git ingest{% endblock %}</title>

    <script src="https://cdn.tailwindcss.com"></script>
    <script src="/static/js/utils.js"></script>
    <script src="/static/js/snow.js"></script>
    <style>
        #snow-canvas {
            position: fixed;
            top: 0;
            left: 0;
            pointer-events: none;
            z-index: 10;
            width: 100%;
            height: 100%;
        }
    </style>
    {% block extra_head %}{% endblock %}
</head>

<body class="bg-[#FFFDF8] min-h-screen flex flex-col">
    <canvas id="snow-canvas"></canvas>
    {% include 'components/navbar.jinja' %}

    <!-- Main content wrapper -->
    <main class="flex-1 w-full">
        <div class="max-w-4xl mx-auto px-4 py-8">
            {% block content %}{% endblock %}
        </div>
    </main>

    {% include 'components/footer.jinja' %}
    
    {% block extra_scripts %}{% endblock %}
</body>
</html> 

================================================
File: /src/templates/github.jinja
================================================
{% extends "base.jinja" %}

{% block content %}
{% if error_message %}
<div class="mb-6 p-4 bg-red-50 border border-red-200 rounded-lg text-red-700" id="error-message"
    data-message="{{ error_message }}">
    {{ error_message }}
</div>
{% endif %}

{% with is_index=true, show_examples=false %}
    {% include 'components/github_form.jinja' %}
{% endwith %}

{% if loading %}
<div class="relative mt-10">
    <div class="w-full h-full absolute inset-0 bg-black rounded-xl translate-y-2 translate-x-2"></div>
    <div class="bg-[#fafafa] rounded-xl border-[3px] border-gray-900 p-6 relative z-20 flex flex-col items-center space-y-4">
        <div class="loader border-8 border-[#fff4da] border-t-8 border-t-[#ffc480] rounded-full w-16 h-16 animate-spin"></div>
        <p class="text-lg font-bold text-gray-900">Loading...</p>
    </div>
</div>
{% endif %}

{% include 'components/result.jinja' %}
{% endblock content %}

{% block extra_scripts %}
<script>
    document.addEventListener('DOMContentLoaded', function() {
        const urlInput = document.getElementById('input_text');
        const form = document.getElementById('ingestForm');
        if (urlInput && urlInput.value.trim() && form) {
            // Wait for stars to be loaded before submitting
            waitForStars().then(() => {
                const submitEvent = new SubmitEvent('submit', { 
                    cancelable: true,
                    bubbles: true
                });
                Object.defineProperty(submitEvent, 'target', {
                    value: form,
                    enumerable: true
                });
                handleSubmit(submitEvent, false);
            });
        }
    });

    function waitForStars() {
        return new Promise((resolve) => {
            const checkStars = () => {
                const stars = document.getElementById('github-stars');
                if (stars && stars.textContent !== '0') {
                    resolve();
                } else {
                    setTimeout(checkStars, 10);
                }
            };
            checkStars();
        });
    }
</script>
{% endblock extra_scripts %}

================================================
File: /src/templates/components/github_form.jinja
================================================
<div class="relative">
    <div class="w-full h-full absolute inset-0 bg-gray-900 rounded-xl translate-y-2 translate-x-2"></div>
    <div class="rounded-xl relative z-20 pl-8 sm:pl-10 pr-8 sm:pr-16 py-8 border-[3px] border-gray-900 bg-[#fff4da]">
        <img src="https://cdn.devdojo.com/images/january2023/shape-1.png"
            class="absolute md:block hidden left-0 h-[4.5rem] w-[4.5rem] bottom-0 -translate-x-full ml-3">

        <form class="flex md:flex-row flex-col w-full h-full justify-center items-stretch space-y-5 md:space-y-0 md:space-x-5"
            id="ingestForm" onsubmit="handleSubmit(event{% if is_index %}, true{% endif %})">
            <div class="relative w-full h-full">
                <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0 z-10"></div>
                <input type="text" name="input_text" id="input_text" placeholder="https://github.com/..."
                    value="{{ github_url if github_url else '' }}"
                    required
                    class="border-[3px] w-full relative z-20 border-gray-900 placeholder-gray-600 text-lg font-medium focus:outline-none py-3.5 px-6 rounded">
            </div>
            <div class="relative w-auto flex-shrink-0 h-full group">
                <div class="w-full h-full rounded bg-gray-800 translate-y-1 translate-x-1 absolute inset-0 z-10"></div>
                <button type="submit"
                    class="py-3.5 rounded px-6 group-hover:-translate-y-px group-hover:-translate-x-px ease-out duration-300 z-20 relative w-full border-[3px] border-gray-900 font-medium bg-[#ffc480] tracking-wide text-lg flex-shrink-0 text-gray-900">
                    Ingest
                </button>
            </div>
            <input type="hidden" name="pattern_type" value="exclude">
            <input type="hidden" name="pattern" value="">
        </form>

        <div class="mt-4 relative z-20 flex flex-wrap gap-4 items-start">
            <!-- Pattern selector -->
            <div class="w-[200px] sm:w-[250px] mr-9 mt-4">
                <div class="relative">
                    <div class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0 z-10"></div>
                    <div class="flex relative z-20 border-[3px] border-gray-900 rounded bg-white">
                        <div class="relative flex items-center">
                            <select id="pattern_type" 
                                    name="pattern_type"
                                    class="w-21 py-2 pl-2 pr-6 appearance-none bg-[#e6e8eb] focus:outline-none border-r-[3px] border-gray-900">
                                <option value="exclude" {% if pattern_type == 'exclude' or not pattern_type %}selected{% endif %}>Exclude</option>
                                <option value="include" {% if pattern_type == 'include' %}selected{% endif %}>Include</option>
                            </select>
                            <svg class="absolute right-2 w-4 h-4 pointer-events-none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
                                <polyline points="6 9 12 15 18 9"/>
                            </svg>
                        </div>
                        <input type="text" 
                               id="pattern"
                               name="pattern" 
                               placeholder="*.md, src/ "
                               value="{{ pattern if pattern else '' }}"
                               class=" py-2 px-2 bg-[#E8F0FE] focus:outline-none w-full">
                    </div>
                </div>
            </div>

            <div class="w-[200px] sm:w-[200px] mt-3">
                <label for="file_size" class="block text-gray-700 mb-1">Include files under: <span id="size_value" class="font-bold">50kb</span></label>
                <input type="range" 
                    id="file_size" 
                    name="max_file_size"
                    min="0" 
                    max="500"
                    required
                    value="{{ default_file_size }}"
                    class="w-full h-3 
                           bg-[#FAFAFA]
                           bg-no-repeat 
                           bg-[length:50%_100%] 
                           bg-[#ebdbb7]
                           appearance-none 
                           border-[3px] 
                           border-gray-900 
                           rounded-sm 
                           focus:outline-none 
                           bg-gradient-to-r from-[#FE4A60] to-[#FE4A60] 
                           [&::-webkit-slider-thumb]:w-5 
                           [&::-webkit-slider-thumb]:h-7 
                           [&::-webkit-slider-thumb]:appearance-none 
                           [&::-webkit-slider-thumb]:bg-white 
                           [&::-webkit-slider-thumb]:rounded-sm 
                           [&::-webkit-slider-thumb]:cursor-pointer 
                           [&::-webkit-slider-thumb]:border-solid
                           [&::-webkit-slider-thumb]:border-[3px]
                           [&::-webkit-slider-thumb]:border-gray-900
                           [&::-webkit-slider-thumb]:shadow-[3px_3px_0_#000]
                           
                           ">
            </div>
        </div>

        {% if show_examples %}
        <!-- Example repositories section -->
        <div class="mt-4">
            <p class="opacity-70 mb-1">Try these example repositories:</p>
            <div class="flex flex-wrap gap-2">
                {% for example in examples %}
                <button onclick="submitExample('{{ example.url }}')"
                    class="px-4 py-1 bg-[#EBDBB7] hover:bg-[#FFC480] text-gray-900 rounded transition-colors duration-200 border-[3px] border-gray-900 relative hover:-translate-y-px hover:-translate-x-px">
                    {{ example.name }}
                    </button>
                    {% endfor %}
            </div>
        </div>
        {% endif %}
    </div>
</div> 

================================================
File: /src/templates/components/navbar.jinja
================================================
<script>
    function formatStarCount(count) {
        if (count >= 1000) {
            return (count / 1000).toFixed(1) + 'k';
        }
        return count.toString();
    }

    async function fetchGitHubStars() {
        try {
            const response = await fetch('https://api.github.com/repos/cyclotruc/gitingest');
            const data = await response.json();
            const starCount = data.stargazers_count;

            document.getElementById('github-stars').textContent = formatStarCount(starCount);
        } catch (error) {
            console.error('Error fetching GitHub stars:', error);
            document.getElementById('github-stars').parentElement.style.display = 'none';
        }
    }

    fetchGitHubStars();
</script>

<header class="sticky top-0 bg-[#FFFDF8] border-b-[3px] border-gray-900 z-50">
    <div class="max-w-4xl mx-auto px-4">
        <div class="flex justify-between items-center h-16">
            <!-- Logo -->
            <div class="flex items-center gap-4">
                <h1 class="text-2xl font-bold tracking-tight">
                    <a href="/" class="hover:opacity-80 transition-opacity">
                        <span class="text-gray-900">Git</span><span class="text-[#fca847]">ingest</span>
                    </a>
                </h1>
            </div>

            <!-- Navigation with updated styling -->
            <nav class="flex items-center space-x-6">
                <a href="/api" class="text-gray-900 hover:-translate-y-0.5 transition-transform">API</a>
                <div class="flex items-center gap-2">
                    <a href="https://github.com/cyclotruc/gitingest" target="_blank" rel="noopener noreferrer"
                        class="text-gray-900 hover:-translate-y-0.5 transition-transform flex items-center gap-1.5">
                        <svg class="w-4 h-4" fill="currentColor" viewBox="0 0 24 24" aria-hidden="true">
                            <path fill-rule="evenodd"
                                d="M12 2C6.477 2 2 6.484 2 12.017c0 4.425 2.865 8.18 6.839 9.504.5.092.682-.217.682-.483 0-.237-.008-.868-.013-1.703-2.782.605-3.369-1.343-3.369-1.343-.454-1.158-1.11-1.466-1.11-1.466-.908-.62.069-.608.069-.608 1.003.07 1.531 1.032 1.531 1.032.892 1.53 2.341 1.088 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.113-4.555-4.951 0-1.093.39-1.988 1.029-2.688-.103-.253-.446-1.272.098-2.65 0 0 .84-.27 2.75 1.026A9.564 9.564 0 0112 6.844c.85.004 1.705.115 2.504.337 1.909-1.296 2.747-1.027 2.747-1.027.546 1.379.202 2.398.1 2.651.64.7 1.028 1.595 1.028 2.688 0 3.848-2.339 4.695-4.566 4.943.359.309.678.92.678 1.855 0 1.338-.012 2.419-.012 2.747 0 .268.18.58.688.482A10.019 10.019 0 0022 12.017C22 6.484 17.522 2 12 2z"
                                clip-rule="evenodd"></path>
                        </svg>
                        GitHub
                    </a>
                    <div class="flex items-center text-sm text-gray-600">
                        <svg class="w-4 h-4 text-[#ffc480] mr-1" fill="currentColor" viewBox="0 0 20 20">
                            <path
                                d="M9.049 2.927c.3-.921 1.603-.921 1.902 0l1.07 3.292a1 1 0 00.95.69h3.462c.969 0 1.371 1.24.588 1.81l-2.8 2.034a1 1 0 00-.364 1.118l1.07 3.292c.3.921-.755 1.688-1.54 1.118l-2.8-2.034a1 1 0 00-1.175 0l-2.8 2.034c-.784.57-1.838-.197-1.539-1.118l1.07-3.292a1 1 0 00-.364-1.118L2.98 8.72c-.783-.57-.38-1.81.588-1.81h3.461a1 1 0 00.951-.69l1.07-3.292z" />
                        </svg>
                        <span id="github-stars">0</span>
                    </div>
                </div>
            </nav>
        </div>
    </div>
</header>

================================================
File: /src/templates/components/result.jinja
================================================
{% if result %}
            <div class="mt-10" data-results>
                <div class="relative">
                    <div class="w-full h-full absolute inset-0 bg-gray-900 rounded-xl translate-y-2 translate-x-2"></div>
                    <div class="bg-[#fafafa] rounded-xl border-[3px] border-gray-900 p-6 relative z-20 space-y-6">
                        <!-- Summary and Directory Structure -->
                        <div class="grid grid-cols-1 md:grid-cols-12 gap-6">
                            <!-- Summary Column -->
                            <div class="md:col-span-5">
                                <div class="flex justify-between items-center mb-4 py-2">
                                    <h3 class="text-lg font-bold text-gray-900">Summary</h3>
                                </div>

                                
                                <div class="relative">
                                    <div
                                        class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
                                    </div>
                                    <textarea
                                        class="w-full h-[160px] p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-none focus:outline-none relative z-10"
                                        readonly>{{ summary }}</textarea>
                                </div>
                                {% if ingest_id %}                                
                                <div class="relative mt-4 inline-block group">
                                    <div
                                        class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
                                    </div>
                                    <a href="/download/{{ ingest_id }}"
                                        class="inline-flex items-center px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10">
                                        <svg class="w-4 h-4 mr-2" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                                            <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2"
                                                d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4" />
                                        </svg>
                                        Download
                                    </a>
                                </div>
                                <div class="relative mt-4 inline-block group ml-4">
                                    <div
                                        class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
                                    </div>
                                    <button onclick="copyFullDigest()"
                                        class="inline-flex items-center px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10">
                                        <svg class="w-4 h-4 mr-2" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                                            <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2"
                                                d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
                                        </svg>
                                        Copy all
                                    </button>
                                </div>
                                {% endif %}

                                
                            </div>

                            <!-- Directory Structure Column -->
                            <div class="md:col-span-7">
                                <div class="flex justify-between items-center mb-4">
                                    <h3 class="text-lg font-bold text-gray-900">Directory Structure</h3>
                                    <div class="relative group">
                                        <div
                                            class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
                                        </div>
                                        <button onclick="copyText('directory-structure')"
                                            class="px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10 flex items-center gap-2">
                                            <svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                                                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2"
                                                    d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
                                            </svg>
                                            Copy
                                        </button>
                                    </div>
                                </div>
                                <div class="relative">
                                    <div
                                        class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
                                    </div>
                                    <textarea
                                        class="directory-structure w-full p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-y focus:outline-none relative z-10 h-[215px]"
                                        readonly>{{ tree }}</textarea>
                                </div>
                            </div>
                        </div>

                        <!-- Full Digest -->
                        <div>
                            <div class="flex justify-between items-center mb-4">
                                <h3 class="text-lg font-bold text-gray-900">Files Content</h3>
                                <div class="relative group">
                                    <div
                                        class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
                                    </div>
                                    <button onclick="copyText('result-text')"
                                        class="px-4 py-2 bg-[#ffc480] border-[3px] border-gray-900 text-gray-900 rounded group-hover:-translate-y-px group-hover:-translate-x-px transition-transform relative z-10 flex items-center gap-2">
                                        <svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                                            <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2"
                                                d="M8 5H6a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2v-1M8 5a2 2 0 002 2h2a2 2 0 002-2M8 5a2 2 0 012-2h2a2 2 0 012 2m0 0h2a2 2 0 012 2v3m2 4H10m0 0l3-3m-3 3l3 3" />
                                        </svg>
                                        Copy
                                    </button>
                                </div>
                            </div>
                            <div class="relative">
                                <div
                                    class="w-full h-full rounded bg-gray-900 translate-y-1 translate-x-1 absolute inset-0">
                                </div>
                                <textarea
                                    class="result-text w-full p-4 bg-[#fff4da] border-[3px] border-gray-900 rounded font-mono text-sm resize-y focus:outline-none relative z-10"
                                    style="min-height: {{ '600px' if content else 'calc(100vh-800px)' }}"
                                    readonly>{{ content }}</textarea>
                            </div>
                        </div>
                    </div>
                </div>
    </div>
{% endif %}

================================================
File: /src/templates/components/footer.jinja
================================================
<footer class="w-full border-t-[3px] border-gray-900 mt-auto">
    <div class="max-w-4xl mx-auto px-4 py-4">
        <div class="flex justify-center items-center text-gray-900 text-sm">
            <div class="flex flex-col items-center">
                <div class="flex items-center">
                    made with ❤️ by
                    <a href="https://bsky.app/profile/yasbaltrine.bsky.social" target="_blank" rel="noopener noreferrer"
                        class="ml-1 hover:underline">
                        @rom2
                    </a>
                </div>
                <a href="https://discord.gg/zerRaGK9EC" target="_blank" rel="noopener noreferrer"
                    class="mt-1 hover:underline flex items-center">
                    <svg class="w-4 h-4 mr-1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512">
                        <path fill="currentColor" d="M524.531,69.836a1.5,1.5,0,0,0-.764-.7A485.065,485.065,0,0,0,404.081,32.03a1.816,1.816,0,0,0-1.923.91,337.461,337.461,0,0,0-14.9,30.6,447.848,447.848,0,0,0-134.426,0,309.541,309.541,0,0,0-15.135-30.6,1.89,1.89,0,0,0-1.924-.91A483.689,483.689,0,0,0,116.085,69.137a1.712,1.712,0,0,0-.788.676C39.068,183.651,18.186,294.69,28.43,404.354a2.016,2.016,0,0,0,.765,1.375A487.666,487.666,0,0,0,176.02,479.918a1.9,1.9,0,0,0,2.063-.676A348.2,348.2,0,0,0,208.12,430.4a1.86,1.86,0,0,0-1.019-2.588,321.173,321.173,0,0,1-45.868-21.853,1.885,1.885,0,0,1-.185-3.126c3.082-2.309,6.166-4.711,9.109-7.137a1.819,1.819,0,0,1,1.9-.256c96.229,43.917,200.41,43.917,295.5,0a1.812,1.812,0,0,1,1.924.233c2.944,2.426,6.027,4.851,9.132,7.16a1.884,1.884,0,0,1-.162,3.126,301.407,301.407,0,0,1-45.89,21.83,1.875,1.875,0,0,0-1,2.611,391.055,391.055,0,0,0,30.014,48.815,1.864,1.864,0,0,0,2.063.7A486.048,486.048,0,0,0,610.7,405.729a1.882,1.882,0,0,0,.765-1.352C623.729,277.594,590.933,167.465,524.531,69.836ZM222.491,337.58c-28.972,0-52.844-26.587-52.844-59.239S193.056,219.1,222.491,219.1c29.665,0,53.306,26.82,52.843,59.239C275.334,310.993,251.924,337.58,222.491,337.58Zm195.38,0c-28.971,0-52.843-26.587-52.843-59.239S388.437,219.1,417.871,219.1c29.667,0,53.307,26.82,52.844,59.239C470.715,310.993,447.538,337.58,417.871,337.58Z"/>
                    </svg>
                    Discord
                </a>
            </div>
        </div>
    </div>
</footer>

================================================
File: /src/templates/index.jinja
================================================
{% extends "base.jinja" %}

{% block extra_head %}
<script>
    function submitExample(repoName) {
        const input = document.getElementById('input_text');
        input.value = repoName;
        input.focus();
    }
</script>
{% endblock %}

{% block content %}
<div class="mb-12">
    <div
        class="relative w-full mx-auto flex sm:flex-row flex-col justify-center items-start sm:items-center">
        <svg class="h-auto w-16 sm:w-20 md:w-24 flex-shrink-0 p-2 md:relative sm:absolute lg:absolute left-0 lg:-translate-x-full lg:ml-32 md:translate-x-10 sm:-translate-y-16 md:-translate-y-0 -translate-x-2 lg:-translate-y-10"
            viewBox="0 0 91 98" fill="none" xmlns="http://www.w3.org/2000/svg">
            <path
                d="m35.878 14.162 1.333-5.369 1.933 5.183c4.47 11.982 14.036 21.085 25.828 24.467l5.42 1.555-5.209 2.16c-11.332 4.697-19.806 14.826-22.888 27.237l-1.333 5.369-1.933-5.183C34.56 57.599 24.993 48.496 13.201 45.114l-5.42-1.555 5.21-2.16c11.331-4.697 19.805-14.826 22.887-27.237Z"
                fill="#FE4A60" stroke="#000" stroke-width="3.445"></path>
            <path
                d="M79.653 5.729c-2.436 5.323-9.515 15.25-18.341 12.374m9.197 16.336c2.6-5.851 10.008-16.834 18.842-13.956m-9.738-15.07c-.374 3.787 1.076 12.078 9.869 14.943M70.61 34.6c.503-4.21-.69-13.346-9.49-16.214M14.922 65.967c1.338 5.677 6.372 16.756 15.808 15.659M18.21 95.832c-1.392-6.226-6.54-18.404-15.984-17.305m12.85-12.892c-.41 3.771-3.576 11.588-12.968 12.681M18.025 96c.367-4.21 3.453-12.905 12.854-14"
                stroke="#000" stroke-width="2.548" stroke-linecap="round"></path>
        </svg>
        <h1
            class="text-4xl sm:text-5xl sm:pt-20 lg:pt-5 md:text-6xl lg:text-7xl font-bold tracking-tighter w-full inline-block text-left md:text-center relative">
            Prompt-friendly <br>codebase&nbsp;
        </h1>
        <svg class="w-16 lg:w-20 h-auto lg:absolute flex-shrink-0 right-0 bottom-0 md:block hidden translate-y-10 md:translate-y-20 lg:translate-y-4 lg:-translate-x-12 -translate-x-10"
            viewBox="0 0 92 80" fill="none" xmlns="http://www.w3.org/2000/svg">
            <path
                d="m35.213 16.953.595-5.261 2.644 4.587a35.056 35.056 0 0 0 26.432 17.33l5.261.594-4.587 2.644A35.056 35.056 0 0 0 48.23 63.28l-.595 5.26-2.644-4.587a35.056 35.056 0 0 0-26.432-17.328l-5.261-.595 4.587-2.644a35.056 35.056 0 0 0 17.329-26.433Z"
                fill="#5CF1A4" stroke="#000" stroke-width="2.868" class=""></path>
            <path
                d="M75.062 40.108c1.07 5.255 1.072 16.52-7.472 19.54m7.422-19.682c1.836 2.965 7.643 8.14 16.187 5.121-8.544 3.02-8.207 15.23-6.971 20.957-1.97-3.343-8.044-9.274-16.588-6.254M12.054 28.012c1.34-5.22 6.126-15.4 14.554-14.369M12.035 28.162c-.274-3.487-2.93-10.719-11.358-11.75C9.104 17.443 14.013 6.262 15.414.542c.226 3.888 2.784 11.92 11.212 12.95"
                stroke="#000" stroke-width="2.319" stroke-linecap="round"></path>
        </svg>
    </div>
    <p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-8">
        Turn any Git repository into a simple text ingest of its codebase.   
    </p>
    <p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-0">
        This is useful for feeding a codebase into any LLM.
    </p>
    <p class="text-gray-600 text-lg max-w-2xl mx-auto text-center mt-2">
        You can also replace 'hub' with 'ingest' in any Github URL
    </p>
</div>

{% if error_message %}
<div class="mb-6 p-4 bg-red-50 border border-red-200 rounded-lg text-red-700" id="error-message"
    data-message="{{ error_message }}">
    {{ error_message }}
</div>
{% endif %}

{% with is_index=true, show_examples=true %}
    {% include 'components/github_form.jinja' %}
{% endwith %}

{% include 'components/result.jinja' %}




{% endblock %}

================================================
File: /src/main.py
================================================
import os
from dotenv import load_dotenv

from fastapi import FastAPI, Request
from fastapi.templating import Jinja2Templates
from fastapi.responses import HTMLResponse, FileResponse, Response
from fastapi.staticfiles import StaticFiles
from starlette.middleware.trustedhost import TrustedHostMiddleware
from api_analytics.fastapi import Analytics
from slowapi import _rate_limit_exceeded_handler
from slowapi.errors import RateLimitExceeded

from server_utils import limiter
from routers import download, dynamic, index


load_dotenv()

app = FastAPI()
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)

app.mount("/static", StaticFiles(directory="static"), name="static")
app.add_middleware(Analytics, api_key=os.getenv('API_ANALYTICS_KEY'))

# Define the default allowed hosts
default_allowed_hosts = ["gitingest.com", "*.gitingest.com", "localhost"]

# Fetch allowed hosts from the environment variable or use the default
allowed_hosts = os.getenv("ALLOWED_HOSTS")
if allowed_hosts:
    allowed_hosts = allowed_hosts.split(",")
else:
    allowed_hosts = default_allowed_hosts

app.add_middleware(TrustedHostMiddleware, allowed_hosts=allowed_hosts)
templates = Jinja2Templates(directory="templates")

@app.get("/health")
async def health_check():
    return {"status": "healthy"}

@app.head("/")
async def head_root():
    """Mirror the headers and status code of the index page"""
    return HTMLResponse(
        content=None,
        headers={
            "content-type": "text/html; charset=utf-8"
        }
    )
    
@app.get("/api/", response_class=HTMLResponse)
@app.get("/api", response_class=HTMLResponse)
async def api_docs(request: Request):
    return templates.TemplateResponse(
        "api.jinja", {"request": request}
    )

@app.get('/favicon.ico')
async def favicon():
    return FileResponse('static/favicon.ico')

@app.get("/robots.txt")
async def robots():
    return FileResponse('static/robots.txt')

app.include_router(index)
app.include_router(download)
app.include_router(dynamic)

================================================
File: /src/gitingest/ingest.py
================================================
import asyncio
import shutil
from typing import Union, List
from pathlib import Path

from .ingest_from_query import ingest_from_query
from .clone import clone_repo
from .parse_query import parse_query

def ingest(source: str, max_file_size: int = 10 * 1024 * 1024, include_patterns: Union[List[str], str] = None, exclude_patterns: Union[List[str], str] = None, output: str = None) -> str:
    try:
        query = parse_query(source, max_file_size, False, include_patterns, exclude_patterns)        
        if query['url']:
            asyncio.run(clone_repo(query))
        
        summary, tree, content = ingest_from_query(query)

        if output:
            with open(f"{output}", "w") as f:
                f.write(tree + "\n" + content)

        return summary, tree, content
    finally:
        # Clean up the temporary directory if it was created
        if query['url']:
            # Get parent directory two levels up from local_path (../tmp)
            cleanup_path = str(Path(query['local_path']).parents[1])
            shutil.rmtree(cleanup_path, ignore_errors=True)

================================================
File: /src/gitingest/ingest_from_query.py
================================================
import os
from fnmatch import fnmatch
from typing import Dict, List, Union
import tiktoken


MAX_FILE_SIZE = 10 * 1024 * 1024  # 10MB
MAX_DIRECTORY_DEPTH = 20  # Maximum depth of directory traversal
MAX_FILES = 10000  # Maximum number of files to process
MAX_TOTAL_SIZE_BYTES = 500 * 1024 * 1024  # 500MB


def should_include(path: str, base_path: str, include_patterns: List[str]) -> bool:
    rel_path = path.replace(base_path, "").lstrip(os.sep)
    include = False
    for pattern in include_patterns:
        if fnmatch(rel_path, pattern):
            include = True
    return include

def should_exclude(path: str, base_path: str, ignore_patterns: List[str]) -> bool:
    rel_path = path.replace(base_path, "").lstrip(os.sep)
    for pattern in ignore_patterns:
        if pattern == '':
            continue
        if fnmatch(rel_path, pattern):
            return True
    return False

def is_safe_symlink(symlink_path: str, base_path: str) -> bool:
    """Check if a symlink points to a location within the base directory."""
    try:
        target_path = os.path.realpath(symlink_path)
        base_path = os.path.realpath(base_path)
        return os.path.commonpath([target_path, base_path]) == base_path
    except (OSError, ValueError):
        # If there's any error resolving the paths, consider it unsafe
        return False

def is_text_file(file_path: str) -> bool:
    """Determines if a file is likely a text file based on its content."""
    try:
        with open(file_path, 'rb') as file:
            chunk = file.read(1024)
        return not bool(chunk.translate(None, bytes([7, 8, 9, 10, 12, 13, 27] + list(range(0x20, 0x100)))))
    except IOError:
        return False

def read_file_content(file_path: str) -> str:
    try:
        with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
            return f.read()
    except Exception as e:
        return f"Error reading file: {str(e)}"

def scan_directory(path: str, query: dict, seen_paths: set = None, depth: int = 0, stats: Dict = None) -> Dict:
    """Recursively analyzes a directory and its contents with safety limits."""
    if seen_paths is None:
        seen_paths = set()
    if stats is None:
        stats = {"total_files": 0, "total_size": 0}

    if depth > MAX_DIRECTORY_DEPTH:
        print(f"Skipping deep directory: {path} (max depth {MAX_DIRECTORY_DEPTH} reached)")
        return None

    if stats["total_files"] >= MAX_FILES:
        print(f"Skipping further processing: maximum file limit ({MAX_FILES}) reached")
        return None

    if stats["total_size"] >= MAX_TOTAL_SIZE_BYTES:
        print(f"Skipping further processing: maximum total size ({MAX_TOTAL_SIZE_BYTES/1024/1024:.1f}MB) reached")
        return None

    real_path = os.path.realpath(path)
    if real_path in seen_paths:
        print(f"Skipping already visited path: {path}")
        return None
    seen_paths.add(real_path)

    result = {
        "name": os.path.basename(path),
        "type": "directory",
        "size": 0,
        "children": [],
        "file_count": 0,
        "dir_count": 0,
        "path": path,
        "ignore_content": False
    }

    ignore_patterns = query['ignore_patterns']
    base_path = query['local_path']
    include_patterns = query['include_patterns']

    try:
        for item in os.listdir(path):
            item_path = os.path.join(path, item)

            if should_exclude(item_path, base_path, ignore_patterns):
                continue

            is_file = os.path.isfile(item_path)
            if is_file and query['include_patterns']:
                if not should_include(item_path, base_path, include_patterns):
                    result["ignore_content"] = True
                    continue

            # Handle symlinks
            if os.path.islink(item_path):
                if not is_safe_symlink(item_path, base_path):
                    print(f"Skipping symlink that points outside base directory: {item_path}")
                    continue
                real_path = os.path.realpath(item_path)
                if real_path in seen_paths:
                    print(f"Skipping already visited symlink target: {item_path}")
                    continue

                if os.path.isfile(real_path):
                    file_size = os.path.getsize(real_path)
                    if stats["total_size"] + file_size > MAX_TOTAL_SIZE_BYTES:
                        print(f"Skipping file {item_path}: would exceed total size limit")
                        continue

                    stats["total_files"] += 1
                    stats["total_size"] += file_size

                    if stats["total_files"] > MAX_FILES:
                        print(f"Maximum file limit ({MAX_FILES}) reached")
                        return result

                    is_text = is_text_file(real_path)
                    content = read_file_content(real_path) if is_text else "[Non-text file]"

                    child = {
                        "name": item,
                        "type": "file",
                        "size": file_size,
                        "content": content,
                        "path": item_path
                    }
                    result["children"].append(child)
                    result["size"] += file_size
                    result["file_count"] += 1

                elif os.path.isdir(real_path):
                    subdir = scan_directory(real_path, query, seen_paths, depth + 1, stats)
                    if subdir and (not include_patterns or subdir["file_count"] > 0):
                        subdir["name"] = item
                        subdir["path"] = item_path
                        result["children"].append(subdir)
                        result["size"] += subdir["size"]
                        result["file_count"] += subdir["file_count"]
                        result["dir_count"] += 1 + subdir["dir_count"]
                continue

            if os.path.isfile(item_path):
                file_size = os.path.getsize(item_path)
                if stats["total_size"] + file_size > MAX_TOTAL_SIZE_BYTES:
                    print(f"Skipping file {item_path}: would exceed total size limit")
                    continue

                stats["total_files"] += 1
                stats["total_size"] += file_size

                if stats["total_files"] > MAX_FILES:
                    print(f"Maximum file limit ({MAX_FILES}) reached")
                    return result

                is_text = is_text_file(item_path)
                content = read_file_content(item_path) if is_text else "[Non-text file]"

                child = {
                    "name": item,
                    "type": "file",
                    "size": file_size,
                    "content": content,
                    "path": item_path
                }
                result["children"].append(child)
                result["size"] += file_size
                result["file_count"] += 1

            elif os.path.isdir(item_path):
                subdir = scan_directory(item_path, query, seen_paths, depth + 1, stats)
                if subdir and (not include_patterns or subdir["file_count"] > 0):
                    result["children"].append(subdir)
                    result["size"] += subdir["size"]
                    result["file_count"] += subdir["file_count"]
                    result["dir_count"] += 1 + subdir["dir_count"]

    except PermissionError:
        print(f"Permission denied: {path}")

    return result

def extract_files_content(query: dict, node: Dict, max_file_size: int, files: List = None) -> List[Dict]:
    """Recursively collects all text files with their contents."""
    if files is None:
        files = []

    if node["type"] == "file" and node["content"] != "[Non-text file]":
        content = node["content"]
        if node["size"] > max_file_size:
            content = None

        files.append({
            "path": node["path"].replace(query['local_path'], ""),
            "content": content,
            "size": node["size"]
        })
    elif node["type"] == "directory":
        for child in node["children"]:
            extract_files_content(query, child, max_file_size, files)
    return files

def create_file_content_string(files: List[Dict]) -> str:
    """Creates a formatted string of file contents with separators."""
    output = ""
    separator = "=" * 48 + "\n"

    for file in files:
        if not file['content']:
            continue
        output += separator
        output += f"File: {file['path']}\n"
        output += separator
        output += f"{file['content']}\n\n"

    return output

def create_summary_string(query: dict, nodes: Dict, files: List[Dict]) -> str:
    """Creates a summary string with file counts and content size."""
    if "user_name" in query:
        summary = f"Repository: {query['user_name']}/{query['repo_name']}\n"
    else:
        summary = f"Repository: {query['slug']}\n"
    summary += f"Files analyzed: {nodes['file_count']}\n"

    if 'subpath' in query and query['subpath'] != '/':
        summary += f"Subpath: {query['subpath']}\n"
    if 'commit' in query and query['commit']:
        summary += f"Commit: {query['commit']}\n"
    elif 'branch' in query and query['branch'] != 'main' and query['branch'] != 'master' and query['branch']:
        summary += f"Branch: {query['branch']}\n"
    return summary

def create_tree_structure(query: dict, node: Dict, prefix: str = "", is_last: bool = True) -> str:
    """Creates a tree-like string representation of the file structure."""
    tree = ""
    if not node["name"]:
        node["name"] = query['slug']

    if node["name"]:
        current_prefix = "└── " if is_last else "├── "
        name = node["name"] + "/" if node["type"] == "directory" else node["name"]
        tree += prefix + current_prefix + name + "\n"
    if node["type"] == "directory":
        # Adjust prefix only if we added a node name
        new_prefix = prefix + ("    " if is_last else "│   ") if node["name"] else prefix
        children = node["children"]
        for i, child in enumerate(children):
            tree += create_tree_structure(query, child, new_prefix, i == len(children) - 1)

    return tree

def generate_token_string(context_string: str) -> str:
    """Returns the number of tokens in a text string."""
    formatted_tokens = ""
    try:
        encoding = tiktoken.get_encoding("cl100k_base", )
        total_tokens = len(encoding.encode(context_string, disallowed_special=()))
        
    except Exception as e:
        print(e)
        return None
    if total_tokens > 1000000:
        formatted_tokens = f"{total_tokens/1000000:.1f}M"
    elif total_tokens > 1000:
        formatted_tokens = f"{total_tokens/1000:.1f}k"
    else:
        formatted_tokens = f"{total_tokens}"
    return formatted_tokens

def ingest_single_file(path: str, query: dict) -> Dict:
    if not os.path.isfile(path):
        raise ValueError(f"Path {path} is not a file")

    file_size = os.path.getsize(path)
    is_text = is_text_file(path)
    if not is_text:
        raise ValueError(f"File {path} is not a text file")

    content = read_file_content(path)
    if file_size > query['max_file_size']:
        content = "[Content ignored: file too large]"

    file_info = {
        "path": path.replace(query['local_path'], ""),
        "content": content,
        "size": file_size
    }

    summary = (
        f"Repository: {query['user_name']}/{query['repo_name']}\n"
        f"File: {os.path.basename(path)}\n"
        f"Size: {file_size:,} bytes\n"
        f"Lines: {len(content.splitlines()):,}\n"
    )

    files_content = create_file_content_string([file_info])
    tree = "Directory structure:\n└── " + os.path.basename(path)

    formatted_tokens = generate_token_string(files_content)
    if formatted_tokens:
        summary += f"\nEstimated tokens: {formatted_tokens}"
    return (summary, tree, files_content)

def ingest_directory(path: str, query: dict) -> Dict:
    nodes = scan_directory(path, query)
    files = extract_files_content(query, nodes, query['max_file_size'])
    summary = create_summary_string(query, nodes, files)
    tree = "Directory structure:\n" + create_tree_structure(query, nodes)
    files_content = create_file_content_string(files)

    formatted_tokens = generate_token_string(tree + files_content)
    if formatted_tokens:
        summary += f"\nEstimated tokens: {formatted_tokens}"
    return (summary, tree, files_content)

def ingest_from_query(query: dict) -> Dict:
    """Main entry point for analyzing a codebase directory or single file."""
    path = f"{query['local_path']}{query['subpath']}"
    if not os.path.exists(path):
        raise ValueError(f"{query['slug']} cannot be found")

    if query.get('type') == 'blob':
        return ingest_single_file(path, query)
    else:
        return ingest_directory(path, query)



================================================
File: /src/gitingest/tests/conftest.py
================================================
import os
import sys

# Get the absolute path of the project root directory (one level up from tests)
project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# Add both the project root and src directory to PYTHONPATH
sys.path.insert(0, project_root)
sys.path.insert(0, os.path.join(project_root, 'src')) 

================================================
File: /src/gitingest/tests/test_clone.py
================================================
import pytest
from clone import clone_repo, check_repo_exists
from unittest.mock import patch, AsyncMock

@pytest.mark.asyncio
async def test_clone_repo_with_commit():
    query = {
        'commit': 'a' * 40,  # Simulating a valid commit hash
        'branch': 'main',
        'url': 'https://github.com/user/repo',
        'local_path': '/tmp/repo'
    }
    
    with patch('clone.check_repo_exists', return_value=True) as mock_check:
        with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
            mock_process = AsyncMock()
            mock_process.communicate.return_value = (b'output', b'error')
            mock_exec.return_value = mock_process
            
            await clone_repo(query)
            mock_check.assert_called_once_with(query['url'])
            assert mock_exec.call_count == 2  # Clone and checkout calls

@pytest.mark.asyncio
async def test_clone_repo_without_commit():
    query = {
        'commit': None,
        'branch': 'main',
        'url': 'https://github.com/user/repo',
        'local_path': '/tmp/repo'
    }
    
    with patch('clone.check_repo_exists', return_value=True) as mock_check:
        with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
            mock_process = AsyncMock()
            mock_process.communicate.return_value = (b'output', b'error')
            mock_exec.return_value = mock_process
            
            await clone_repo(query)
            mock_check.assert_called_once_with(query['url'])
            assert mock_exec.call_count == 1  # Only clone call

@pytest.mark.asyncio
async def test_clone_repo_nonexistent_repository():
    query = {
        'commit': None,
        'branch': 'main',
        'url': 'https://github.com/user/nonexistent-repo',
        'local_path': '/tmp/repo'
    }
    
    with patch('gitingest.clone.check_repo_exists', return_value=False) as mock_check:
        with pytest.raises(ValueError, match="Repository not found"):
            await clone_repo(query)
            mock_check.assert_called_once_with(query['url'])

@pytest.mark.asyncio
async def test_check_repo_exists():
    url = "https://github.com/user/repo"
    
    with patch('asyncio.create_subprocess_exec', new_callable=AsyncMock) as mock_exec:
        mock_process = AsyncMock()
        
        # Test existing repository
        mock_process.returncode = 0
        mock_exec.return_value = mock_process
        assert await check_repo_exists(url) is True
        
        # Test non-existing repository
        mock_process.returncode = 1
        mock_exec.return_value = mock_process
        assert await check_repo_exists(url) is False

================================================
File: /src/gitingest/tests/test_ingest.py
================================================
import pytest
from src.gitingest.ingest_from_query import (
    scan_directory,
    extract_files_content,
)

# Test fixtures
@pytest.fixture
def sample_query():
    return {
        'user_name': 'test_user',
        'repo_name': 'test_repo',
        'local_path': '/tmp/test_repo',
        'subpath': '/',
        'branch': 'main',
        'commit': None,
        'max_file_size': 1000000,
        'slug': 'test_user/test_repo',
        'ignore_patterns': ['*.pyc', '__pycache__', '.git'],
        'include_patterns': None,
        'pattern_type': 'exclude'
        
    }

@pytest.fixture
def temp_directory(tmp_path):
    # Creates the following structure:
    # test_repo/
    # ├── file1.txt
    # ├── file2.py
    # └── src/
    # |   ├── subfile1.txt
    # |   └── subfile2.py
    # |   └── subdir/
    # |       └── file_subdir.txt
    # |       └── file_subdir.py
    # └── dir1/
    # |   └── file_dir1.txt
    # └── dir2/
    #     └── file_dir2.txt
    
    test_dir = tmp_path / "test_repo"
    test_dir.mkdir()
    
    # Root files
    (test_dir / "file1.txt").write_text("Hello World")
    (test_dir / "file2.py").write_text("print('Hello')")
    
    # src directory and its files
    src_dir = test_dir / "src"
    src_dir.mkdir()
    (src_dir / "subfile1.txt").write_text("Hello from src")
    (src_dir / "subfile2.py").write_text("print('Hello from src')")
    
    # src/subdir and its files
    subdir = src_dir / "subdir"
    subdir.mkdir()
    (subdir / "file_subdir.txt").write_text("Hello from subdir")
    (subdir / "file_subdir.py").write_text("print('Hello from subdir')")
    
    # dir1 and its file
    dir1 = test_dir / "dir1"
    dir1.mkdir()
    (dir1 / "file_dir1.txt").write_text("Hello from dir1")
    
    # dir2 and its file
    dir2 = test_dir / "dir2"
    dir2.mkdir()
    (dir2 / "file_dir2.txt").write_text("Hello from dir2")
    
    return test_dir

def test_scan_directory(temp_directory, sample_query):
    result = scan_directory(
        str(temp_directory),
        query=sample_query
    )
    
    assert result['type'] == 'directory'
    assert result['file_count'] == 8  # All .txt and .py files
    assert result['dir_count'] == 4   # src, src/subdir, dir1, dir2
    assert len(result['children']) == 5  # file1.txt, file2.py, src, dir1, dir2

def test_extract_files_content(temp_directory, sample_query):
    nodes = scan_directory(
        str(temp_directory),
        query=sample_query
    )
    
    files = extract_files_content(sample_query, nodes, max_file_size=1000000)
    assert len(files) == 8  # All .txt and .py files
    
    # Check for presence of key files
    paths = [f['path'] for f in files]
    assert any('file1.txt' in p for p in paths)
    assert any('subfile1.txt' in p for p in paths)
    assert any('file2.py' in p for p in paths)
    assert any('subfile2.py' in p for p in paths)
    assert any('file_subdir.txt' in p for p in paths)
    assert any('file_dir1.txt' in p for p in paths)
    assert any('file_dir2.txt' in p for p in paths)



# TODO: test with include patterns: ['*.txt']
# TODO: test with wrong include patterns: ['*.qwerty']


#single folder patterns
# TODO: test with include patterns: ['src/*']
# TODO: test with include patterns: ['/src/*']
# TODO: test with include patterns: ['/src/']
# TODO: test with include patterns: ['/src*']

#multiple patterns
# TODO: test with multiple include patterns: ['*.txt', '*.py']
# TODO: test with multiple include patterns: ['/src/*', '*.txt']
# TODO: test with multiple include patterns: ['/src*', '*.txt']






================================================
File: /src/gitingest/tests/test_parse_query.py
================================================
import pytest
from gitingest.parse_query import parse_query, parse_url, DEFAULT_IGNORE_PATTERNS


def test_parse_url_valid():
    test_cases = [
        "https://github.com/user/repo",
        "https://gitlab.com/user/repo", 
        "https://bitbucket.org/user/repo"
    ]
    for url in test_cases:
        result = parse_url(url)
        assert result["user_name"] == "user"
        assert result["repo_name"] == "repo"
        assert result["url"] == url

def test_parse_url_invalid():
    url = "https://only-domain.com"
    with pytest.raises(ValueError, match="Invalid repository URL"):
        parse_url(url)

def test_parse_query_basic():
    test_cases = [
        "https://github.com/user/repo",
        "https://gitlab.com/user/repo"
    ]
    for url in test_cases:
        result = parse_query(url, max_file_size=50, from_web=True, ignore_patterns='*.txt')
        assert result["user_name"] == "user"
        assert result["repo_name"] == "repo"
        assert result["url"] == url
        assert "*.txt" in result["ignore_patterns"]

def test_parse_query_include_pattern():
    url = "https://github.com/user/repo"
    result = parse_query(url, max_file_size=50, from_web=True, include_patterns='*.py')
    assert result["include_patterns"] == ["*.py"]
    assert result["ignore_patterns"] == DEFAULT_IGNORE_PATTERNS

def test_parse_query_invalid_pattern():
    url = "https://github.com/user/repo"
    with pytest.raises(ValueError, match="Pattern.*contains invalid characters"):
        parse_query(url, max_file_size=50, from_web=True, include_patterns='*.py;rm -rf')

================================================
File: /src/gitingest/__init__.py
================================================
from .ingest_from_query import ingest_from_query
from .clone import clone_repo
from .parse_query import parse_query
from .ingest import ingest

__all__ = ["ingest_from_query", "clone_repo", "parse_query", "ingest"]

================================================
File: /src/gitingest/cli.py
================================================
import os
import pathlib
import click

from gitingest.ingest import ingest
from gitingest.ingest_from_query import MAX_FILE_SIZE
from gitingest.parse_query import DEFAULT_IGNORE_PATTERNS

def normalize_pattern(pattern: str) -> str:
    pattern = pattern.strip()
    pattern = pattern.lstrip(os.sep)
    if pattern.endswith(os.sep):
        pattern += "*"
    return pattern

@click.command()
@click.argument('source', type=str, required=True)
@click.option('--output', '-o', default=None, help='Output file path (default: <repo_name>.txt in current directory)')
@click.option('--max-size', '-s', default=MAX_FILE_SIZE, help='Maximum file size to process in bytes')
@click.option('--exclude-pattern', '-e', multiple=True, help='Patterns to exclude')
@click.option('--include-pattern', '-i', multiple=True, help='Patterns to include')
def main(source, output, max_size, exclude_pattern, include_pattern):
    """Analyze a directory and create a text dump of its contents."""
    try:
        # Combine default and custom ignore patterns
        exclude_patterns = list(exclude_pattern)
        include_patterns = list(set(include_pattern))
        
        if not output:
            output = "digest.txt"
        summary, tree, content = ingest(source, max_size, include_patterns, exclude_patterns, output=output)
            
        click.echo(f"Analysis complete! Output written to: {output}")
        click.echo("\nSummary:")
        click.echo(summary)
        
    except Exception as e:
        click.echo(f"Error: {str(e)}", err=True)
        raise click.Abort()

if __name__ == '__main__':
    main() 

================================================
File: /src/gitingest/utils.py
================================================

## Async Timeout decorator
import asyncio
import functools
from typing import TypeVar, Callable

T = TypeVar("T")

class AsyncTimeoutError(Exception):
    """Raised when an async operation exceeds its timeout limit."""
    pass

def async_timeout(seconds: int = 10):
    def decorator(func: Callable[..., T]) -> Callable[..., T]:
        @functools.wraps(func)
        async def wrapper(*args, **kwargs) -> T:
            try:
                return await asyncio.wait_for(func(*args, **kwargs), timeout=seconds)
            except asyncio.TimeoutError:
                raise AsyncTimeoutError(f"Operation timed out after {seconds} seconds")
        return wrapper
    return decorator

================================================
File: /src/gitingest/parse_query.py
================================================
from typing import List, Union
import uuid
import os 


DEFAULT_IGNORE_PATTERNS = [
    # Python
    '*.pyc', '*.pyo', '*.pyd', '__pycache__', '.pytest_cache', '.coverage',
    '.tox', '.nox', '.mypy_cache', '.ruff_cache', '.hypothesis',
    'poetry.lock', 'Pipfile.lock',
    
    # JavaScript/Node
    'node_modules', 'bower_components', 'package-lock.json', 'yarn.lock',
    '.npm', '.yarn', '.pnpm-store',
    
    # Version control
    '.git', '.svn', '.hg', '.gitignore', '.gitattributes', '.gitmodules',
    
    # Images and media
    '*.svg', '*.png', '*.jpg', '*.jpeg', '*.gif', '*.ico', '*.pdf',
    '*.mov', '*.mp4', '*.mp3', '*.wav',
    
    # Virtual environments
    'venv', '.venv', 'env', '.env', 'virtualenv',
    
    # IDEs and editors
    '.idea', '.vscode', '.vs', '*.swp', '*.swo', '*.swn',
    '.settings', '.project', '.classpath', '*.sublime-*',
    
    # Temporary and cache files
    '*.log', '*.bak', '*.swp', '*.tmp', '*.temp',
    '.cache', '.sass-cache', '.eslintcache',
    '.DS_Store', 'Thumbs.db', 'desktop.ini',
    
    # Build directories and artifacts
    'build', 'dist', 'target', 'out',
    '*.egg-info', '*.egg', '*.whl',
    '*.so', '*.dylib', '*.dll', '*.class',
    
    # Documentation
    'site-packages', '.docusaurus', '.next', '.nuxt',
    
    # Other common patterns
    '*.min.js', '*.min.css',  # Minified files
    '*.map',  # Source maps
    '.terraform', '*.tfstate*',  # Terraform
    'vendor/',  # Dependencies in various languages
]

TMP_BASE_PATH = "../tmp"

def parse_url(url: str) -> dict:
    parsed = {
        "user_name": None,
        "repo_name": None,
        "type": None,
        "branch": None,
        "commit": None,
        "subpath": "/",
        "local_path": None,
        "url": None,
        "slug": None,
        "id": None,
    }
    
    url = url.split(" ")[0]
    if not url.startswith('https://'):
        url = 'https://' + url
        
    # Extract domain and path
    url_parts = url.split('/')
    domain = url_parts[2]
    path_parts = url_parts[3:]
    
    if len(path_parts) < 2:
        raise ValueError("Invalid repository URL. Please provide a valid Git repository URL.")
        
    parsed["user_name"] = path_parts[0]
    parsed["repo_name"] = path_parts[1]
    
    # Keep original URL format
    parsed["url"] = f"https://{domain}/{parsed['user_name']}/{parsed['repo_name']}"
    parsed['slug'] = f"{parsed['user_name']}-{parsed['repo_name']}"
    parsed["id"] = str(uuid.uuid4())
    parsed["local_path"] = f"{TMP_BASE_PATH}/{parsed['id']}/{parsed['slug']}"

    if len(path_parts) > 3:
        parsed["type"] = path_parts[2]
        parsed["branch"] = path_parts[3]
        if len(parsed['branch']) == 40 and all(c in '0123456789abcdefABCDEF' for c in parsed['branch']):
            parsed["commit"] = parsed['branch']
            
        parsed["subpath"] = "/" + "/".join(path_parts[4:])
    return parsed

def normalize_pattern(pattern: str) -> str:
    pattern = pattern.strip()
    pattern = pattern.lstrip(os.sep)
    if pattern.endswith(os.sep):
        pattern += "*"
    return pattern

def parse_patterns(pattern: Union[List[str], str]) -> List[str]:
    if isinstance(pattern, list):
        pattern = ",".join(pattern)

    for p in pattern.split(","):
        if not all(c.isalnum() or c in "-_./+*" for c in p.strip()):
            raise ValueError(f"Pattern '{p}' contains invalid characters. Only alphanumeric characters, dash (-), underscore (_), dot (.), forward slash (/), plus (+), and asterisk (*) are allowed.")
    patterns = [normalize_pattern(p) for p in pattern.split(",")]
    return patterns

def override_ignore_patterns(ignore_patterns: List[str], include_patterns: List[str]) -> List[str]:
    for pattern in include_patterns:
        if pattern in ignore_patterns:
            ignore_patterns.remove(pattern)
    return ignore_patterns


def parse_path(path: str) -> dict:
    
    query = {
        "local_path": os.path.abspath(path),
        "slug": os.path.basename(os.path.dirname(path)) + "/" + os.path.basename(path),
        "subpath": "/",
        "id": str(uuid.uuid4()),
        "url": None,
    }
    return query

def parse_query(source: str, max_file_size: int, from_web: bool, include_patterns: Union[List[str], str] = None, ignore_patterns: Union[List[str], str] = None) -> dict:
    if from_web:
        query = parse_url(source)
    else:
        if source.startswith("https://") or "github.com" in source:
            query = parse_url(source)
        else:
            query = parse_path(source)
    query['max_file_size'] = max_file_size

    if ignore_patterns and ignore_patterns != "":
        ignore_patterns = DEFAULT_IGNORE_PATTERNS + parse_patterns(ignore_patterns)
    else:
        ignore_patterns = DEFAULT_IGNORE_PATTERNS
    
    if include_patterns and include_patterns != "":
        include_patterns = parse_patterns(include_patterns)
        ignore_patterns = override_ignore_patterns(ignore_patterns, include_patterns)
    else: 
        include_patterns = None
    
    query['ignore_patterns'] = ignore_patterns
    query['include_patterns'] = include_patterns
    
    return query



================================================
File: /src/gitingest/clone.py
================================================
import asyncio
from typing import Tuple

from gitingest.utils import async_timeout

CLONE_TIMEOUT = 20

async def check_repo_exists(url: str) -> bool:
    proc = await asyncio.create_subprocess_exec(
        "git",
        "ls-remote",
        url,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    await proc.communicate()
    return proc.returncode == 0

@async_timeout(CLONE_TIMEOUT)
async def clone_repo(query: dict) -> str:
    if not await check_repo_exists(query['url']):
        raise ValueError("Repository not found, make sure it is public")
        
    if query['commit']:
        proc = await asyncio.create_subprocess_exec(
            "git", 
            "clone",
            "--single-branch",
            query['url'],
            query['local_path'],
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
        )
        stdout, stderr = await proc.communicate()
        
        proc = await asyncio.create_subprocess_exec(
            "git",
            "-C",
            query['local_path'],
            "checkout",
            query['branch'],
            stdout=asyncio.subprocess.PIPE, 
            stderr=asyncio.subprocess.PIPE,
        )
        stdout, stderr = await proc.communicate()
    elif query['branch'] != 'main' and query['branch'] != 'master' and query['branch']:
        proc = await asyncio.create_subprocess_exec(
            "git",
            "clone", 
            "--depth=1",
            "--single-branch",
            "--branch",
            query['branch'],
            query['url'],
            query['local_path'],
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
        )
    else:
        proc = await asyncio.create_subprocess_exec(
            "git",
            "clone",
            "--depth=1",
            "--single-branch",
            query['url'],
            query['local_path'],
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
        )
        
    stdout, stderr = await proc.communicate()
    
    return stdout, stderr