Skip to content

Latest commit

 

History

History
1213 lines (959 loc) · 31.4 KB

usage.adoc

File metadata and controls

1213 lines (959 loc) · 31.4 KB

Git Provider Sync: Technical Documentation

1. Introduction

Git Provider Sync is a tool that enables repository synchronization across different Git Providers and storage. This document provides guidance on its configuration and usage.

1.1 Key Features

Git Provider Sync enables:

  • Multi-source Git provider configurations

  • Multiple target types per source:

    • Git provider (user/organization)

    • Archive storage

    • Directory storage

2. Getting Started

2.1 Installation

  1. Choose an installation method:

    • Pre-packaged distributions

    • Manual build

    • Refer to installation guide for detailed instructions

2.2 Basic Setup

  1. Create configuration file

  2. Validate configuration:

gitprovidersync print

+ 3. Initialize synchronization:

gitprovidersync sync

3. Command Line Interface

3.1 Basic Usage

  • Example: CLI Help Output

Display available options:
gitprovidersync --help
gpshelp
  • CLI Sync Command Help Output

Display available sync options:
gitprovidersync sync --help
gpssynchelp
  • Access manual (package installations only):

man gitprovidersync

3.2 CLI Usage Examples

Configuration Display

Print the current configuration, using a non-default configuration file location

gitprovidersync print --config-file /custom/path/config.yaml

Advanced Synchronization

Sync, using force push, fetch recent update (3 hours to now)

gitprovidersync --force-push --from='-3h' --alphanumhyph-name --config-file /path/config.yaml

4. Configuration Specific

4.1 Configuration Sources

Priority order (highest to lowest):

  1. Environment variables

  2. .env file

  3. Configuration file

  4. XDG_CONFIG_HOME

4.2 File Location Hierarchy

  1. Default filename: gitprovidersync.yaml

  2. Search locations:

    • Binary execution directory

    • $XDG_CONFIG_HOME/gitprovidersync

    • Custom location (via --config-file)

4.3 Environment Variables

  • Prefix: GPS_

  • Format: Uppercase with underscores

  • Example: GPS_GITPROVIDERSYNC_SOURCE_PROVIDER=envconfprovider

4.4 Configuration Examples

Simple: A sync from a GitHub to an GitLab-instance, in the simplest way. All public repos. Default github.com and gitlab.com domains.
gitprovidersync:
  environment:
    mysimplesourceonfig:
        provider_type: github
        owner: itiquette

        mirror:
          agitlabtarget:
            provider_type: gitlab
            owner: <a-gitlab-user>
            owner_type: user
            auth:
              token: <a-gitlab-token-with-create-and-write-access>
Extended: A sync from a GitHub to a GitLab-instance, archive dir, and a dir. Because we can. NOTE DONT PUT SECRETS IN CONFIG FILES OTHER THAN FOR TESTING, use envs or alike
gitprovidersync:
  environment:
    asourceconf:
      provider_type: github
      domain: custom.github.domain
      owner: itiquette
      auth:
        token: <a-github-token-so-we-avoid-rate-limiting-etc>
      repositories:
        include: org-feature-test-repo # we only want one repo from this org

      mirrors:
        agitlabtarget:
          provider_type: gitlab
          domain: custom.gitlab.domain
          owner: <a-gitlab-user>
          auth:
            token: <a-gitlab-token-with-create-and-write-access>

        tartargetexample:
          providertype: archive
          path: /tmp/myarcdir

        directoryexample:
          provider_type: directory
          path: /tmp/mydir
Git-Binary: A sync from gitlab to github. special case, uses gitbinary, ssh and custom ssh-client to push ssh through githubs tls endpoint. It also uses corkscrew on the host.
gitprovidersync:
  environment:
    aconfigname:
      provider_type: gitlab
      owner: hanklank
      auth:
        token: <a-gitlab-token>
      repositories:
        include: org-feature-test-repo # we only want one repo from this org
      use_git_binary: true

      mirrors:
        agithubtarget:
          provider_type: github
          owner: <a-gitlab-user>
          owner_type: user
          auth:
            token: <a-gitlab-token-with-create-and-write-access>
            ssh_command: ssh -vvv -o ProxyCommand="corkscrew <proxyserver> <proxyserverport> %h %p" -o ConnectTimeout=10 -p 443
            rewrite_ssh_url_from: [email protected]:
            rewrite_ssh_url_to: ssh://[email protected]:443/
            type: ssh

          #This SSH command uses corkscrew to tunnel SSH through an HTTP proxy. It's set to maximum verbosity (-vvv), uses a 10-second connection timeout, and connects to the SSH server on port 443. The ProxyCommand option specifies the proxy server and port, with %h and %p as placeholders for the SSH server host and port

4.5 Advanced Configuration

Using a host Git Binary instead of the underlying Go Git Library

In certain cases you might like to use the underlying host Git Binary instead of the built in Go Git-library. For example, If you would like to customize the underlying SSH-client with GIT_SSH_COMMAND. Go Git does not currently support SSH-client customisation.

ℹ️
Only use this if you really have to (for example, you might want to use the SSHCommand option).

5. Provider-Specific

5.1 Authentication Methods

  • Default: Use Token Access

  • Alternative for non-API access: SSH with sshagent

GitLab API

GitLab API has the following Authentication methods:

Auth Type Use Case Security Level Expiration Scope Control

BasicAuth

Development/Testing

Low

N/A

Full Access

JobToken Docs

CI/CD Pipelines

High

Job Duration

Job-specific

OAuthToken

Third-party Apps

High

Configurable

Granular

PrivateToken Personal Token Group Token

Automation/Scripts

High

Never

Granular

Git Provider Sync currently supports Private Token/Group Token.

GitHub API

Auth Type Use Case Security Level Expiration Scope Control

Personal Access Token (Classic) Docs

General Automation/Scripts

High

Never (unless revoked)

Repository/Org level

Fine-grained PAT Docs

Specific repository access

Very High

Required (max 1 year)

Repository specific

GITHUB_TOKEN Docs

GitHub Actions

High

Job Duration

Repository scoped

OAuth App Token

Third-party Apps

High

Configurable with refresh

User-authorized scopes

Git Provider Sync currently supports Classic/Fine Grained and GITHUB_TOKEN.

Gitea API

Auth Type Use Case Security Level Expiration Scope Control

Basic Auth

Development/Testing

Low

N/A

Full Access

Access Token Docs

General Automation/Scripts

High

Never (unless revoked)

Repository/Organization level

OAuth2 Token

Third-party Apps

High

Configurable

Granular scopes

Internal Token Docs

Admin operations

Very High

Never

Full system access

Action Token Docs

Actions (CI/CD)

High

Job Duration

Repository scoped

To generate an access token for a Gitea user with creation rights:

curl -H "Content-Type: application/json" -d '{"name":"<tokenname>","scopes":["write:organization","write:repository","read:user","write:user"]}' -u user:password https://<giteahost>/api/v1/users/<username>/tokens

5.2 Provider Rate Limits

Platform Unauthenticated With Token Notes

GitHub.com

60 requests/hour

  • Basic Token: 5,000/hour

  • 83x increase with token

GitLab.com

60 requests/hour

  • Free: 2,000/hour

  • Premium: 2,500/hour

  • Ultimate: 3,000/hour

  • Tier determines limit

  • CI tokens have separate limits

Gitea.com

10 requests/minute

20 requests/minute

  • 2x increase with token

  • Per-minute rather than per-hour

  • GitHub: Authentication method matters

  • GitLab: User tier matters

  • Gitea: Simple doubling with any authentication

  • GitHub: Authentication method matters

  • GitLab: User tier matters

  • Gitea: Simple doubling with any authentication

6 Target-Specific

6.1 Directory Target

  • Contains working copy repositories from the source

  • Default: Check out all remote branches locally and keep original origin remote

Configuration example:

...
..
    localtar:
      provider_type: directory
      path: <full/path/to/directory/where/repositories/go>

6.2 Compressed Archive (tar.gz) Target

  • Contains tar.gz files of bare repositories

  • Adds a timestamp prefix to allow multiple re-runs

Configuration example:

...
..
    localtar:
      provider_type: archive
      path: <full/path/to/directory/where/tar/archives/go>

7. CI Deployment Examples

A few examples of how you can run Git Provider Syns in various CI/CD environments.

7.1 GitLab CI Example

This GitLab CI configuration defines two jobs for repository synchronization:

  1. scheduled-sync: Runs automatically on schedule, using the GitLab CI schedule feature

  2. manual_sync: Triggered manually with confirmation prompt

Both jobs use a container image specified and share a common script template that:

  • Prints the configuration

  • Executes a sync run with specified time limit and force push

The configuration uses YAML anchors for DRY script configuration.

The configuration example includes variables for:

  • Source provider settings (GitLab user/group, included repositories, time limit)

  • Mirror target configuration (provider type, domain, owner details)

  • Authentication token (expected as CI/CD secret)

A simple GitLab CI example
# This GitLab CI example shows two GitLab CI jobs.
# A regular job, to run on schedule.
# A manual job, to be run manually from the GitLab CI UI.

variables:
  IMAGE:
    description: "Path to a container image"
    value: "ghcr.io/itiquette/gitprovidersync:v0.0.20"

  # Source provider configuration example

  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_PROVIDER_TYPE:
    description: "Source provider for the configuration (e.g., gitlab, github,gitea see docs)"
    value: "gitlab"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_OWNER:
    description: "Username for the source provider"
    value: "auser"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_OWNER_TYPE:
    description: "User type for the source provider (user,group)"
    value: "auser"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_REPOSITORIES_INCLUDE:
    description: "Comma-separated list of repositories to include (of empty, all are fetched)"
    value: "areponame,anotherreponame"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_ACTIVE_FROM_LIMIT:
    description: "How old changes to be considered, golang string time duration format."
    value: "-30h"

  # Mirror target configuration example
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_MIRRORS_EXAMPLETARGET1_PROVIDER_TYPE:
    description: "Mirror target for the configuration (e.g., gitlab, github, directory, tar, directory etc)"
    value: "gitlab"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_MIRRORS_EXAMPLETARGET1_DOMAIN:
    description: "Domain of the mirror target, if any (e.g., gitlab.com, yourgitlab.domainname)"
    value: "gitlab.com"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_MIRRORS_EXAMPLETARGET1_OWNER:
    description: "Mirror target group or user"
    value: "ausername"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_MIRRORS_EXAMPLETARGET1_OWNER_TYPE:
    description: "Mirror target group or user type (group,user)"
    value: "user"
  GPS_GITPROVIDERSYNC_ENV_EXAMPLECONF1_MIRRORS_EXAMPLETARGET1_AUTH_TOKEN: <asecrettoken>

  # Also, you would need a masked token/secret with correct write acccess
  # Never commit that, use it as a secret from you CI/CD env

# A GitLab CI Template for the script part
.git-provider-sync-script-template: &sync_script
  image:
    name: $IMAGE
    entrypoint: [""]
  variables:
    GIT_STRATEGY: none # Disable Git fetching
  script:
    - gitprovidersync print
    - gitprovidersync sync --active-from-limit $ACTIVE_FROM_LIMIT --force-push

# Regular job that only runs on schedule.
# To schedule the GitLab CI job to run at regular intervals,
# you can use GitLab's CI/CD pipelines schedule feature.
# This allows you to trigger pipelines at specific times using cron syntax.
scheduled-sync:
  <<: *sync_script
  only:
    - schedules

# Manual job extending the Template above
manual_sync:
  when: manual
  manual_confirmation: "Are you sure you want to start a git-provider-sync run?"
  except:
    - schedules
  <<: *sync_script

7.2 GitHub Action

The GitHub Actions Example configuration defines two jobs for repository synchronization:

  1. scheduled-sync: Runs on cron schedule

  2. manual_sync: Triggered via workflow_dispatch with customizable inputs

Configuration variables include:

  • Source settings (provider type, domain, owner details)

  • Mirror target settings (provider type, domain, owner)

  • Auth tokens (stored as GitHub Secrets)

  • Allows dynamic input parameters via UI

  • Has concurrency control to cancel in-progress jobs

A GitHub Action example
# This GitHub Action example shows two GitHub Action jobs.
# A regular job, to run on a schedule.
# A manual job, to be run manually from the GitHub Action UI.

name: Git Provider Sync

defaults:
  run:
    env:
      PREFIX: GPS_GITPROVIDERSYNC_DEFAULTENV_EXAMPLE
      TARGET1: GPS_GITPROVIDERSYNC_DEFAULTENV_EXAMPLE_MIRRORS_TARGET1

on:
  schedule:
    - cron: "0 */2 * * *" # Adjust cron expression as needed for scheduling.
        # This example would run every other hour.
  workflow_dispatch: # Allows manual triggering from the GitHub Actions UI, and inputs: allows for GUI input dynamically
      # # https://github.com/orgs/community/discussions/26324
      #  IMAGE:
      #    description: "Path to container image"
      #    required: true
      #    default: "path/to/gitprovidersync/image:version"
    inputs:
        # source provider configuration
        provider_type:
          description: "Source provider type"
          required: true
          default: "gitlab"
        domain:
          description: "Source provider domain"
          required: true
          default: "gitlab.com"
        owner:
          description: "Source owner"
          required: true
          default: "auser"
        owner_type:
          description: "Source owner type (default: group (organisation))"
          required: false
          default: "group"
        repositories_include:
          description: "Repositories to include (if not given, fetch all)"
          required: false
          default: "areponame,anotherreponame"
        active_from_limit:
          description: "Change age limit"
          required: true
          default: "-30000h"

        # mirror target configuration
        mirror_provider_type:
          description: "Mirror target type"
          required: true
          default: "gitlab"
        mirror_domain:
          description: "Mirror target domain"
          required: true
          default: "gitlab.com"
        mirror_owner:
          description: "Mirror target owner"
          required: true
          default: "ausername"
        mirror_owner_type:
          description: "Mirror target owner type"
          required: false
          default: "user"

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true # cancel ongoing jobs

jobs:
  scheduled-sync:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    environment: test
    container:
      image: "path/to/gitprovidersync/image:version"
    env:
      ${{ env.PREFIX }}_PROVIDER_TYPE: github
      ${{ env.PREFIX }}_DOMAIN: github.com
      ${{ env.PREFIX }}_OWNER: auser
      ${{ env.PREFIX }}_OWNER_TYPE: user
      ${{ env.PREFIX }}_REPOSITORIES_INCLUDE: areponame,anotherreponame
      ${{ env.PREFIX }}_ACTIVE_FROM_LIMIT: "-30000h"
      ${{ env.TARGET1 }}_PROVIDER_TYPE: gitlab
      ${{ env.TARGET1 }}_DOMAIN: gitlab.com
      ${{ env.TARGET1 }}_OWNER: ausername
      ${{ env.TARGET1 }}_OWNER_TYPE: user
      ${{ env.TARGET1 }}_AUTH_TOKEN: ${{ secrets.EXAMPLETARGET1_TOKEN }}
  # You would set secret tokens in GitHub Secrets

    if: github.event_name == 'schedule'
    steps:
      - name: Print configuration
        run: gitprovidersync print

      - name: Sync configuration
        run: gitprovidersync sync --active-from-limit ${{ env.PREFIX }}_${{ ACTIVE_FROM_LIMIT }} --force-push


  manual_sync:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    environment: test
    container:
      # image: ${{ env.IMAGE }} # https://github.com/orgs/community/discussions/26324
      image: "ghcr.io/itiquette/gitprovidersync:v0.0.20"
    env:
      ${{ env.PREFIX }}_PROVIDER_TYPE: ${{ inputs.provider_type }}
      ${{ env.PREFIX }}_DOMAIN: ${{ inputs.domain }}
      ${{ env.PREFIX }}_OWNER: ${{ inputs.owner }}
      ${{ env.PREFIX }}_OWNER_TYPE: ${{ inputs.owner_type }}
      ${{ env.PREFIX }}_REPOSITORIES_INCLUDE: ${{ inputs.repositories_include }}
      ${{ env.PREFIX }}_ACTIVE_FROM_LIMIT: ${{ inputs.active_from_limit }}

      ${{ env.TARGET1 }}_PROVIDER_TYPE: ${{ inputs.target_provider_type }}
      ${{ env.TARGET1 }}_DOMAIN: ${{ inputs.target_domain }}
      ${{ env.TARGET1 }}_OWNER: ${{ inputs.target_owner }}
      ${{ env.TARGET1 }}_OWNER_TYPE: ${{ inputs.target_owner_type }}
      ${{ env.TARGET1 }}_AUTH_TOKEN: ${{ secrets.EXAMPLETARGET1_TOKEN }}
      ${{ env.PREFIX }}_ACTIVE_FROM_LIMIT: ${{ inputs.active_from_limit }}

    if: github.event_name == 'workflow_dispatch'
    steps:
      - name: Print configuration
        run: gitprovidersync print

      - name: Sync configuration
        run: gitprovidersync sync --active-from-limit ${{ env.PREFIX }}_${{ ACTIVE_FROM_LIMIT }} --force-push

7.3 Tekton CI

ℹ️
Tekton doesn’t have native support for scheduled triggers or manual dispatch. Use Tekton Triggers and CronJobs for similar functionality.

The example Kubernetes CronJob defines a scheduled repository synchronization task:

  • Runs every 2 hours using Kubernetes cron scheduling

  • Uses concurrency control to prevent overlapping jobs

  • Employs a retry policy with one attempt and a timeout

  • Maintains minimal job history (1 successful, 1 failed)

  • Environment variables for source/mirror target settings

  • Token authentication via Kubernetes secrets

The configuration uses Kubernetes native features for scheduling and secret management instead of CI-specific constructs.

A Scheduled Run Example using Kubernetes CronJob:
apiVersion: batch/v1
kind: CronJob
metadata:
 name: git-provider-sync-cronjob
spec:
 schedule: "0 */2 * * *"
 concurrencyPolicy: Forbid
 successfulJobsHistoryLimit: 1
 failedJobsHistoryLimit: 1
 jobTemplate:
   spec:
     backoffLimit: 1
     activeDeadlineSeconds: 900
     template:
       spec:
         containers:
           - name: git-provider-sync
             image: ghcr.io/itiquette/gitprovidersync:v0.0.20
             env:
               - name: PREFIX
                 value: GPS_GITPROVIDERSYNC_DEFAULTENV_EXAMPLE
               - name: TARGET1
                 value: GPS_GITPROVIDERSYNC_DEFAULTENV_EXAMPLE_MIRRORS_TARGET1

               - name: $(PREFIX)_PROVIDER_TYPE
                 value: gitlab
               - name: $(PREFIX)_DOMAIN
                 value: gitlab.com
               - name: $(PREFIX)_OWNER
                 value: auser
               - name: $(PREFIX)_OWNER_TYPE
                 value: user
               - name: $(PREFIX)_REPOSITORIES_INCLUDE
                 value: areponame,anotherreponame

               - name: $(TARGET1)_PROVIDER_TYPE
                 value: gitlab
               - name: $(TARGET1)_DOMAIN
                 value: gitlab.com
               - name: $(TARGET1)_OWNER
                 value: ausername
               - name: $(TARGET1)_OWNER_TYPE
                 value: user
               - name: $(TARGET1)_AUTH_TOKEN
                 valueFrom:
                   secretKeyRef:
                     name: git-sync-secrets
                     key: target-token

               - name: ACTIVE_FROM_LIMIT
                 value: "-30000h"

             command:
               - "sh"
               - "-c"
               - |
                 gitprovidersync print && \
                 gitprovidersync sync --active-from-limit $ACTIVE_FROM_LIMIT --force-push

         restartPolicy: Never

8. Troubleshooting and FAQ

8.1 Common Questions

Q: Is dual HTTP/SSH configuration required?

A: No. HTTPS with tokens is recommended for simplicity. SSH support is available but requires additional configuration.

Appendix A: Provider Visibility Mappings

Different providers have slightly different visibility options for repositories. Listed is how they will be synced.

Table 1. GitLab Provider Visibility Mappings
GitLab GitHub Gitea

Public

Public

Public

Internal

Private

Private

Private

Private

Private

Table 2. GitHub Provider Visibility Mappings
GitHub GitLab Gitea

Public

Public

Public

Private

Private

Private

Table 3. Gitea Provider Visibility Mappings
Gitea GitLab GitHub

Public

Public

Public

Private

Private

Private

Limited

Private

Private

Appendix B: Configuration properties table

Property Path Description Required Extra Notes Default Value

gitprovidersync

Root configuration object containing all project configurations

Mandatory

Must contain at least one environment configuration (e.g., production, staging).

gitprovidersync:
  production:
    gitlab-main: ...
    github-source: ...
  staging:
    staging-source: ...

N/A

gitprovidersync.<env>

Environment configuration group (e.g., production, staging)

Mandatory

Must contain at least one source configuration.

production:
  gitlab-main: ...
  github-source: ...

N/A

gitprovidersync.<env>.<source>.provider_type

Git provider type

Mandatory

Must be one of: gitlab, github, gitea.

provider_type: gitlab

N/A

gitprovidersync.<env>.<source>.domain

FQDN Domain name of the Git provider

Optional

Must not include protocol scheme.

domain: gitlab.com

Providertype=DefaultDomain: gitlab=gitlab.com github=github.com gitea=gitea.com

gitprovidersync.<env>.<source>.owner

Repository owner username or group name

Mandatory

Length: 1-255 characters. Can include path segments for groups.

owner: username

N/A

gitprovidersync.<env>.<source>.owner_type

Repository owner type

Mandatory

Must be either 'user' or 'group'

owner_type: user

N/A

gitprovidersync.<env>.<source>.auth.token

Git provider API token

Optional

Required for private repositories and higher API limits.

auth:
  token: ${GIT_TOKEN}

Empty

gitprovidersync.<env>.<source>.auth.http_scheme

Protocol scheme

Optional

Must be http or https. HTTP not recommended for production.

auth:
  http_scheme: https

https

gitprovidersync.<env>.<source>.auth.proxy_url

Proxy URL

Optional

Must be valid URL. Environment vars HTTP_PROXY etc. also supported.

auth:
  proxy_url: http://proxy.company.com:8080

Empty

gitprovidersync.<env>.<source>.auth.cert_dir_path

Directory path for custom certificates

Optional

Must be absolute path and directory must exist.

auth:
  cert_dir_path: /etc/ssl/certs

Empty

gitprovidersync.<env>.<source>.auth.ssh_command

Custom SSH proxy command

Optional

Must start with 'ssh'. Requires use_git_binary: true.

auth:
  ssh_command: ssh -F /custom/ssh/config

Empty

gitprovidersync.<env>.<source>.auth.ssh_url_rewrite_from

Original SSH URL pattern to rewrite

Optional

Must be set if ssh_url_rewrite_to is set.

auth:
  ssh_url_rewrite_from: [email protected]:

Empty

gitprovidersync.<env>.<source>.auth.ssh_url_rewrite_to

Target SSH URL pattern

Optional

Must be set if ssh_url_rewrite_from is set.

auth:
  ssh_url_rewrite_to: git@github-internal:

Empty

gitprovidersync.<env>.<source>.include_forks

Whether to include forked repositories

Optional

Only valid for source provider.

include_forks: false

false

gitprovidersync.<env>.<source>.use_git_binary

Use system git binary instead of go-git library

Optional

Required for SSH proxy command. Git must be installed.

use_git_binary: true

false

gitprovidersync.<env>.<source>.repositories.include

Repositories to include

Optional

Cannot be empty if specified. Supports wildcards.

repositories:
  include:
    - repo1
    - repo2
    - project-*

All repos

gitprovidersync.<env>.<source>.repositories.exclude

Repositories to exclude

Optional

Cannot be empty if specified. Applied after include filter.

repositories:
  exclude:
    - test-*
    - temp-repo

None

gitprovidersync.<env>.<source>.active_from_limit

Age limit for repositories to sync

Optional

Valid Go duration format.

active_from_limit: 24h

Empty

gitprovidersync.<env>.<source>.mirrors

Mirror configurations

Mandatory

Must contain at least one mirror configuration.

mirrors:
  gitlab-mirror:
    provider_type: gitlab
  archive-backup:
    provider_type: archive

N/A

gitprovidersync.<env>.<source>.mirrors.<mirror>.provider_type

Mirror provider type

Mandatory

Must be: gitlab, github, gitea, archive, or directory.

provider_type: gitlab

N/A

gitprovidersync.<env>.<source>.mirrors.<mirror>.path

Directory path for archive/directory type mirrors

Mandatory for archive/directory types

Must be absolute path.

path: /path/to/archives

N/A

gitprovidersync.<env>.<source>.mirrors.<mirrors>.use_git_binary

Use system git binary instead of go-git library

Optional

Required for SSH proxy command. Git must be installed.

use_git_binary: true

false

gitprovidersync.<env>.<source>.mirrors.<mirror>.settings.force_push

Always use force push

Optional

Only valid for Git provider mirrors.

settings:
  force_push: true

false

gitprovidersync.<env>.<source>.mirrors.<mirror>.settings.ignore_invalid_name

Don’t abort on invalid repository names

Optional

Only valid for Git provider mirrors.

settings:
  ignore_invalid_name: true

false

gitprovidersync.<env>.<source>.mirrors.<mirror>.settings.alphanumhyph-name

Clean repository names (alphanumeric only)

Optional

Only valid for Git provider mirrors.

settings:
  alphanumhyph_name: true

false

gitprovidersync.<env>.<source>.mirrors.<mirror>.settings.description_prefix

Description prefix for mirrored repositories

Optional

Max length: 1000 characters.

settings:
  description_prefix: "[Mirror] "

Empty

gitprovidersync.<env>.<source>.mirrors.<mirror>.settings.disabled

Disables as much project settings as possible

Optional

Only valid for Git provider mirrors.

settings:
  disabled: true

true

gitprovidersync.<env>.<source>.mirrors.<mirror>.settings.visibility

Default visibility for target repo

Optional

Only valid for Git provider mirrors.

settings:
  visibility: private

Use source setting

ℹ️

Key Dependencies:

  • SSH Authentication requires:

    • Running SSH agent

    • SSH_AUTH_SOCK environment variable

    • At least one loaded SSH key

  • HTTPS Authentication requires:

    • Valid token for private repositories

    • Valid certificate configuration for custom CAs

  • Path Requirements:

    • All paths must be absolute

    • Directories must exist and have appropriate permissions

    • Certificate directories must be readable

    • Archive/Directory target paths must be writable

Appendix C: Logging

Git Provider Sync follows the Unix philosophy where logs are diagnostics (stderr) and not part of the program’s actual output (stdout).

Levels

  • Use TRACE, DEBUG during development

  • INFO, WARN, ERROR in production

  • FATAL used occasionally for startup errors

  • PANIC is used rarely and will abort the current run

TRACE helps you follow code execution path and program flow.

DEBUG helps you find bug values

INFO shows normal operation

WARN shows things to watch but not errors

ERROR shows actual problems, and oftent they will abort the current run.

FATAL - severe error, will call os.Exit(1)

PANIC - severe error, will call panic()

Tips

  • Log verbosity are given with the --verbosity option.

    gitprovidersync sync --verbosity=TRACE
  • As logs are written to stderr, if you filter with grep, you might want to pipe stderr to stdout

    --verbosity=TRACE 2>&1 >/dev/null | grep 'TRC'
  • For even more verbose information while developing, add caller output with option

    --verbosity-with-caller
  • For output in json, choose output format

    --output-format=json
  • For quiet output, only outputting errors, use option

    --quiet

Appendix D: Advanced: Restoring a Working Copy from a Compressed Archived Bare Git Repository

  1. Unpack the tar.gz file:

    tar -xvzf <path/to/tar.gz-archive> [-C /path/to/target/dir]
  2. Clone the bare repository to get a working copy:

    git clone <path/to/unpacked-bare-git-repository> [/path/to/clone-target/dir]
  3. (Optional) Check out all branches:

    for b in `git branch -r | grep -v -- '->'`; do git branch --track ${b##origin/} $b; done
  4. (Optional) Change the "origin" of your working copy:

    1. Show current remote origin:

      git remote -v
    2. Print original remote origin:

      (cd <path/to/unpacked-bare-git-repository>; git remote -v)
    3. Set new remote origin:

      git remote set-url origin <the origin url shown above>
ℹ️
For HTTPS remote origins, consider using SSH format instead.