A simple Git LFS server implementation in Dart

Overview

A simple Git LFS server implementation in Dart

CI Release

Overview

Git LFS client (git-lfs or git lfs command) requires a dedicated LFS-content server (not the Git server!) in order to download/upload binary files. There will be an issue if you try to use a mirror of a Git repo with LFS-enabled: git-lfs#1338. This Git LFS server implementation solves that problem.

There are other implementations that you can look for if you're not working with Git mirrors.

Quick Reference

Benefit

There are two important reasons to set up a Git server for mirroring with LFS-enabled:

  • To save bandwidth on the connection to the remote server, especially if the repositories are big.
  • Team members don't have access to the remote server, but your Git server can.

Limitations

  • Only download operation is supported. You cannot upload binary files.
  • LFS contents in the mirror are not fetched automatically by this tool.

Quick Start

For the sake of simplicity, I have created a Docker Hub repository with the images ready to use. Assumed that you have the same problem as in git-lfs#1338 (There's a picture in it if you like graphical presentation). Although, for a smoothly experience, I would add some more details:

  • Mirrors will be accessible via SSH, i.e., you can do git clone git@GIT_SERVER:MIRROR_REPO.git, or git clone ssh://git@GIT_SERVER:PORT/MIRROR_REPO.git
  • Self-signed certificate: we need a .key file, a .crt file, a .pem file. Follow this guide to create one.
  • You will need some automation to fetch latest Git references and LFS contents from the far repositories, in case the far repositories are newer than the mirrors.
  • Far remote is LFS-enabled.
  • Near remote has Docker installed.

If all requirements are satisfied, let us proceed step-by-step

  1. Pull the image: docker pull khoa10/amz-git-mirroring:latest
  2. There must be a folder named certificates with 3 files: git-lfs-server.key, git-lfs-server.cert, git-lfs-server.pem. Create a volume to store the certificate:
$ ls certificates
git-lfs-server.crt  git-lfs-server.key  git-lfs-server.pem
$ docker volume create --name git-lfs-server-certs
$ docker run --rm -v $PWD:/source -v git-lfs-server-certs:/dest -w /source alpine cp "certificates/git-lfs-server.key" "certificates/git-lfs-server.crt" "certificates/git-lfs-server.pem" /dest
  1. With your mirrors are located at $INSERT_YOUR_PATH_HERE, create the container:
docker run \
--detach \
--env GIT_LFS_SERVER_URL="https://$INSERT_URL_HERE:8443" \
--mount type=bind,src=$INSERT_YOUR_PATH_HERE,dst=/source \
--mount type=volume,src=git-lfs-server-certs,dst=/etc/git-lfs-server/certificates \
-p 8443:8443 \
-p 2022:22 \
khoa10/amz-git-mirroring

Now, your Git server is available at ssh://git@:2022 and the LFS server is available at https://:8443. Because we're using self-signed certificate, Git client must either ignore TLS verification or trust the certificate you created. To ignore TLS verification:

git config --global http.https://<NEAR_REMOTE>:8443.sslverify false

For better experience, client should configure to download from near and push to far:

git config --global url.ssh://git@<NEAR_REMOTE>:2022.insteadOf ssh://git@<FAR_REMOTE>
git config --global url.ssh://git@<FAR_REMOTE>.pushInsteadOf ssh://git@<NEAR_REMOTE>:2022

License

MIT license

Comments
  • `git-lfs-authenticate` crashes

    `git-lfs-authenticate` crashes

    git-lfs-authenticate crashes with the following backtrace:

    
    [SEVERE] [2022-05-06 13:53:37.672256] [git-lfs-authenticate] [Usage: git-lfs-authenticate {path} {operation}]
    [SEVERE] [2022-05-06 13:53:37.692073] [git-lfs-authentication] [Unknown error!][LateInitializationError: Field '_channel@18358666' has not been initialized.][#0 _channel (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart)
    #1 onExit (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:40:9)
    #2 startAuthenticate (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:47:16)
    #3 main.<anonymous closure> (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:11:11)
    #4 main.<anonymous closure> (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:10:19)
    #5 _rootRun (dart:async/zone.dart:1426:13)
    #6 _CustomZone.run (dart:async/zone.dart:1328:19)
    #7 _runZoned (dart:async/zone.dart:1861:10)
    #8 runZonedGuarded (dart:async/zone.dart:1849:12)
    #9 main (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:10:3)
    #10 _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:295:32)
    #11 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:192:12)
    ]
    Unhandled exception:
    LateInitializationError: Field '_channel@18358666' has not been initialized.
    #0 _channel (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart)
    #1 onExit (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:40:9)
    #2 main.<anonymous closure> (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:15:16)
    #3 main.<anonymous closure> (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart:12:6)
    #4 _RootZone.runBinary (dart:async/zone.dart:1690:54)
    #5 runZonedGuarded.<anonymous closure> (dart:async/zone.dart:1832:18)
    #6 _Zone._processUncaughtError (dart:async/zone.dart:1109:14)
    #7 _CustomZone.handleUncaughtError (dart:async/zone.dart:1312:5)
    #8 Future._propagateToListeners (dart:async/future_impl.dart:680:16)
    #9 Future._completeError (dart:async/future_impl.dart:609:5)
    #10 _completeOnAsyncError (dart:async-patch/async_patch.dart:272:13)
    #11 main.<anonymous closure> (file://${HOME}/.pub-cache/git/git-lfs-server-7ad8ce55d7dd38abe5644928ac3f6cdcc5c2ef35/bin/git_lfs_authenticate.dart)
    <asynchronous suspension>
    
    bug 
    opened by khoa-io 1
  • Workflow CI fails

    Workflow CI fails

    Workflow CI defined in ci.yml always fails for merge commit. The reason is the source branch is deleted. Solution is share source code with arm64 container.

    bug 
    opened by khoa-io 0
  • `README.md` is missing instructions to mount volume `git-lfs-server-certs`

    `README.md` is missing instructions to mount volume `git-lfs-server-certs`

    Unhandled exception:
    
    FileSystemException: Cannot open file, path = '/etc/git-lfs/certificates/git-lfs-server.crt' (OS Error: No such file or directory, errno = 2)
    
    #0      _File.throwIfError (dart:io/file_impl.dart:635)
    
    #1      _File.openSync (dart:io/file_impl.dart:479)
    
    #2      _File.readAsBytesSync (dart:io/file_impl.dart:539)
    
    #3      _SecurityContext.useCertificateChain (dart:io-patch/secure_socket_patch.dart:251)
    
    #4      main (file:///home/runner/work/git-lfs-server/git-lfs-server/bin/git_lfs_server.dart:74)
    
    <asynchronous suspension>
    
    documentation 
    opened by khoa-io 0
  • Only push _versioned_ image to Docker Hub

    Only push _versioned_ image to Docker Hub

    Currently, every push or pull request to main trigger CI. Therefore, a new image will be pushed to Docker Hub with the tag ${{ secrets.DOCKER_HUB_USERNAME }}/amz-git-mirroring:${{ env.BUILD_NUMBER }}-${{ env.SHORT_SHA }}. This will overpopulate the Docker Hub repository. Only versioned image should be pushed with the tag ${{ secrets.DOCKER_HUB_USERNAME }}/amz-git-mirroring:${{ env.BUILD_VERSION }}.

    enhancement 
    opened by khoa-io 0
  • Use environment variable to control logging

    Use environment variable to control logging

    Add the following environment variables to control logging:

    • GIT_LFS_AUTHENTICATE_TRACE: Controls logging of git-lfs-authenticate command.
    • GIT_LFS_SERVER_TRACE: Controls logging of git-lfs-server command.
    • GIT_LFS_AUTH_SERVICE_TRACE: Controls logging of auth-service.
    • GIT_LFS_HTTP_SERVER_TRACE: Controls logging of the http-server.
    ready-for-review 
    opened by khoa-io 0
  • Mirror should fetch latest LFS contents

    Mirror should fetch latest LFS contents

    Git mirrors don't perform fetch themselves. It would be nice if we can fetch from the remote repo every time users fetch from our repo. Not only LFS contents can be fetched, but the Git references as well.

    enhancement backlog 
    opened by khoa-io 0
Releases(v0.0.2-i)
Owner
Hoàng Văn Khoa
Hoàng Văn Khoa
Socketio dart server and client - Full Socket.io implementation using Dart Lang

Getting Started Step 1: Run dart_server.dart Step 2: Android Emulator has proble

Trần Thiên Trọng 1 Jan 23, 2022
Get git information from within the Flutter application

git_info_plus Get git information from within the Flutter application. Usage Dart final String branchName = await GitInfo.branchName; final DateTime l

Akira Aratani 4 May 17, 2022
Access app version and git informations from auto-generated and configurable widgets

This is a proof of concept and WIP Feedback and ideas welcome !! Access your pubspec and git commit informations like versions and commit status from

Robert Felker 15 Jul 7, 2021
Contribute in building the Ultimate GitBook of Developers by creating your git-card.json

Contribute in building the Ultimate GitBook of Developers by creating your git-card.json gitcards The Attractive and Interactive Variant of your GitHu

omega ui 1 Apr 8, 2022
Git+ is your ultimate GitLab mobile app that lets you interact with your projects like as if you were using desktop.

Git+ for GitLab Git+ is your ultimate GitLab mobile app that lets you interact with your projects like as if you were using desktop. Git+ lets you see

Marek Gvora 38 Jan 7, 2023
Get It - Simple direct Service Locator that allows to decouple the interface from a concrete implementation and to access the concrete implementation from everywhere in your App. Maintainer: @escamoteur

❤️ Sponsor get_it This is a simple Service Locator for Dart and Flutter projects with some additional goodies highly inspired by Splat. It can be used

Flutter Community 1k Jan 1, 2023
A simple HTTP server that can serve up any directory, built with Dart

A simple HTTP server that can serve up any directory, built with Dart. Inspired by python -m SimpleHTTPServer. Install Use the dart pub global command

Behruz Hurramov 0 Dec 27, 2021
Dart-com-Shelf - Web server básico feito com dart e shelf, configurações de rotas e conexão com mysql.

A server app built using Shelf, configured to enable running with Docker. This sample code handles HTTP GET requests to / and /echo/<message> Running

Luanzera07 0 Jan 3, 2022
A simple log catch server powered by flutter.

flutter_logcat A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started i

bebe 1 Jan 9, 2022
Petrus Nguyễn Thái Học 193 Dec 29, 2022
A simple dart zeromq implementation/wrapper around the libzmq C++ library

dartzmq A simple dart zeromq implementation/wrapper around the libzmq C++ library Features Currently supported: Creating sockets (pair, pub, sub, req,

Moritz Wirger 18 Dec 29, 2022
A simple and efficient FFT implementation for Dart

fftea A simple and efficient Fast Fourier Transform (FFT) library. FFT converts a time domain signal to the frequency domain, and back again. This is

Liam Appelbe 29 Nov 18, 2022
Pure Dart Client for Nakama Server 🌟🥰🤩

Nakama Flutter Client ?? ?? ?? ?? Nakama is an open-source scalable game server. This is a Flutter client for Nakama written in pure dart and supports

Oliver Brunsmann 57 Dec 6, 2022
A streaming client for the Komga self-hosted comics/manga/BD server targeting Android/iOS written in Dart/Flutter

Klutter A streaming client for the Komga self-hosted comics/manga/BD server targeting Android/iOS written in Dart/Flutter Background This is a project

Mark Winckle 58 Dec 7, 2022
Mentorup App developed using Flutter/Dart and Aqueduct as backend server is an app to help the startups

Mentorup App developed using Flutter/Dart and Aqueduct as backend server is an app to help the startups of various domains initiated by the budding entrepreneurs to get guidance and funding from the mentors and funders respectively.

Ekta Masrani 1 Dec 14, 2021
Mildly encrypted package - An encryption client & server for Dart Native + mobile platforms.

TODO: Put a short description of the package here that helps potential users know whether this package might be useful for them. Features TODO: List w

Andrew Stein 0 Jan 9, 2022
Buildpack dart - A server app built using Shelf, configured to enable running with Docker

Hosted at https://dart-buildpack-demo-ruyjilv5wq-uc.a.run.app/ Buildpack defined

Kevin Moore 12 Dec 15, 2022
Dartness is a progressive dart framework for building efficient and scalable server-side applications

Dartness is a framework for building efficient, scalable dart server-side applications. It provides an easy and quick way to develop modern standalone server.

Ricardo Romero 33 Dec 12, 2022
🦜 Parrot - A progressive Dart framework for building efficient, reliable and scalable server-side applications.

?? Parrot A progressive Dart framework for building efficient, reliable and scalable server-side applications. What is Parrot? Parrot is a Dart framew

Odroe 8 Nov 11, 2022