gitlab netrc

Skip to content

GitLab Next

GitLab FOSS
GitLab FOSS

Collapse sidebar

Closed

Opened by Denis M

Report abuse New issue

BASIC auth not working with go get

Closed

BASIC auth not working with go get

Hi all!

Summary

This is mainly a follow-up bug? of this go get meta feature request (/cc @DouweM) with explicit issues when using gitlab’s subgroup feature, where some part of the repo path is private and thus not publicly accessible. I can confirm, that --header "PRIVATE-TOKEN: ..." is working now, but I was mainly interested in the .netrc support, which is using BASIC auth. Looking at the list of supported authentications there is no explicit BASIC auth support listed, does also not work on the api (3), but git over https does use/allow BASIC auth with the personal access token (6). git is using “plain” https, with a specific middleware supporting basic auth to determine if access is allowed (7) or not (8). Here i think a good quick summary how git is tied with https, curl and .netrc

The big question is, is there a good reason to not support BASIC auth (with the personal access token as password) for go get (surely https is probably a must then) where gitlab’s git is satisfied with it?

Steps to reproduce

The issue can be reproduced on gitlab’s official gitlab, but you’ll have to make your own group/subgroup/privaterepo tree or request access on my subgroup somehow.

  • create a main group (visibility public)
  • create in that main group a subgroup (visibility public)
  • create a project in that group (visibility private)
  • put something in that repo so it is initialized
  • try to go get -v gitlab.com/your/private/project (without .git suffix) and check the recieved go get meta tag

Example Project

As already written, since the key is to access a private repo using your personal access token, I can’t provide an example without intervention from my side (invite to my subgroup/give access to private repo). Easiest would be to create the steps to reproduce for yourself.

What is the current bug behavior?

$ curl -n https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo?go-get=1
<html><head><meta name="go-import" content="gitlab.com/dion-maingroup/dion-subgroup git https://gitlab.com/dion-maingroup/dion-subgroup.git" /></head></html>

Where my .netrc is (where password is my personal access token with scope api)

$ cat .netrc 
machine gitlab.com
login dionysius
password redacted-access-token

What is the expected correct behavior?

$ curl -n https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo?go-get=1
<html><head><meta name="go-import" content="gitlab.com/dion-maingroup/dion-subgroup/privaterepo git https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git" /></head></html>

Relevant logs and/or screenshots

[1] by default my access is unauthorized

$ curl https://gitlab.com/api/v4/version
{"message":"401 Unauthorized"}

[2] using a valid access token I get

$ curl --header "PRIVATE-TOKEN: redacted-access-token" https://gitlab.com/api/v4/version
{"version":"10.6.3-ee","revision":"8b1a92e"}

[3] the basic auth does not work with the api it seems

$ curl -n https://gitlab.com/api/v4/version
{"message":"401 Unauthorized"}

[4] I can download my private git repo (which implicitly uses my .netrc file)

$ git clone https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git
Cloning into 'privaterepo'...
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.

[5] I can’t download that repo without .netrc (to confirm nothing is disrupting auth in any other way)

$ mv ~/.netrc{,.bak}; git clone https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git;
Cloning into 'privaterepo'...
Username for 'https://gitlab.com': ^C

[6] git is using BASIC auth to authenticate me

$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git
15:00:25.402756 git.c:344               trace: built-in: git 'clone' 'https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git'
Cloning into 'privaterepo'...
15:00:25.406662 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git'
*   Trying 52.167.219.168...
[...]
> GET /dion-maingroup/dion-subgroup/privaterepo.git/info/refs?service=git-upload-pack HTTP/1.1
Host: gitlab.com
User-Agent: git/2.16.2
[...]
< HTTP/1.1 401 Unauthorized
< Server: nginx
[...]
< WWW-Authenticate: Basic realm="GitLab"
[...]
* Ignoring the response-body
* Connection #0 to host gitlab.com left intact
* Issue another request to this URL: 'https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git/info/refs?service=git-upload-pack'
* Found bundle for host gitlab.com: 0x8eaac0 [can pipeline]
* Re-using existing connection! (#0) with host gitlab.com
* Connected to gitlab.com (52.167.219.168) port 443 (#0)
* Server auth using Basic with user 'dionysius'
> GET /dion-maingroup/dion-subgroup/privaterepo.git/info/refs?service=git-upload-pack HTTP/1.1
Host: gitlab.com
Authorization: Basic redacted-base64encoding-of-access-token
User-Agent: git/2.16.2
[...]
< HTTP/1.1 200 OK
< Server: nginx
[...] 
* Connection #0 to host gitlab.com left intact
15:00:26.124085 run-command.c:627       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git/'
15:00:26.127328 git.c:344               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git/'
* Found bundle for host gitlab.com: 0x8eaac0 [can pipeline]
* Re-using existing connection! (#0) with host gitlab.com
* Connected to gitlab.com (52.167.219.168) port 443 (#0)
* Server auth using Basic with user 'dionysius'
> POST /dion-maingroup/dion-subgroup/privaterepo.git/git-upload-pack HTTP/1.1
Host: gitlab.com
Authorization: Basic redacted-base64encoding-of-access-token
User-Agent: git/2.16.2
[...]
* upload completely sent off: 165 out of 165 bytes
< HTTP/1.1 200 OK
< Server: nginx
[...]
* Connection #0 to host gitlab.com left intact
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
15:00:26.454472 run-command.c:627       trace: run_command: 'unpack-objects' '--pack_header=2,3'
15:00:26.456683 git.c:344               trace: built-in: git 'unpack-objects' '--pack_header=2,3'
Unpacking objects: 100% (3/3), done.
15:00:26.458918 run-command.c:627       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
15:00:26.460940 git.c:344               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'

[7] git can access on https the refs using basic auth

$ curl -n -v --output - https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git/info/refs?service=git-receive-pack
*   Trying 52.167.219.168...
[...]
* Server auth using Basic with user 'dionysius'
> GET /dion-maingroup/dion-subgroup/privaterepo.git/info/refs?service=git-receive-pack HTTP/1.1
> Host: gitlab.com
> Authorization: Basic redacted-base64encoding-of-access-token
> User-Agent: curl/7.58.0
[...]
< HTTP/1.1 200 OK
< Server: nginx
[...]
001f# service=git-receive-pack
0000009c9e9772ad831ae75388b4837e4a67c35e05f6a0ab refs/heads/masterreport-status delete-refs side-band-64k quiet atomic ofs-delta push-options agent=git/2.14.3
* Connection #0 to host gitlab.com left intact

[8] git can’t access on https the refs without basic auth

$ mv .netrc{,.bak}; curl -n -v --output - https://gitlab.com/dion-maingroup/dion-subgroup/privaterepo.git/info/refs?service=git-receive-pack
* Couldn't find host gitlab.com in the .netrc file; using defaults
*   Trying 52.167.219.168...
[...]
> GET /dion-maingroup/dion-subgroup/privaterepo.git/info/refs?service=git-receive-pack HTTP/1.1
> Host: gitlab.com
> User-Agent: curl/7.58.0
[...] 
< HTTP/1.1 401 Unauthorized
< Server: nginx
[...]
< WWW-Authenticate: Basic realm="GitLab"
[...]
HTTP Basic: Access denied
* Connection #0 to host gitlab.com left intact

Output of checks

This bug happens on GitLab.com

Possible fixes

tbd

Edited by Denis M

 

  • @dionysius Thanks for this detailed report! I think it makes sense to support the same username/password authentication we do on Git-over-HTTP. From an implementation perspective, that would mean running Gitlab::Auth.find_for_git_client if we detect that basic authentication information is provided in Gitlab::Middleware::Go#current_user, like we do in Projects::GitHttpClientController#authenticate_user.

    I don’t know when we’ll be able to prioritize this (/cc @jramsay), but if you’d like to have a try at implementing it yourself I’d be more than happy to guide you through it!

  • Douwe Maan @DouweM added Accepting merge requests ~2278658 authentication backend ~2975006 labels
  • Max Kovgan
    Max Kovgan @max.kovgan ·

 

 

 

 

 

 

  • @DouweM is there any chance this can be implemented by the GitLab team? I think this is quite an important feature for enterprise using Go, I would love to contribute but I don’t have any Ruby coding experience ?

  • Douwe Maan
    Douwe Maan @DouweM ·
    Developer

 

 

 

@krak3n I patched go.rb middleware easily – to skip auth in the intranet (commented out the auth call) No time to start glue other things. The fine grained security means no go get easily. I made an exception for go.rb, it’s data exposure, but the customer is ok with that patch.

The right fix would be to maybe whiltelist clients based on host/netblock and other. and also expose based on proj/subgroup1/submod1/etc. base So that only “authorized” endpoints OR “allowed” projects leak that information.

IF somebody is paying me to do this – I’d do it 😉 the customer preferred the trivial patch.

 

 

 

 

@DouweM Hi, I am implementing this feature in my merge request !23497 (merged) . Would you mind to have a look?

Edited by MortyChoi

 

 

 

 

 

 

 

  • @piccolo-p you might want to provide more detailed info to prove your usage and maybe you found a case I didn’t mentioned.

    Unfortunately I’m not yet on this Gitlab version so I wasn’t able to confirm the resolution myself yet.

  • Douwe Maan
    Douwe Maan @DouweM ·
    Developer

 

 

 

 

  • Hello @DouweM I’v got the same issue here with gitlab 11.8

    I’ve got a .netrc file

    curl -n works as expected but when I try go get on a subgroup, it gets the wrong metadata.

    Did I miss somehting ? Thanks for your help!

    Edited by Alexandre Assouad
  • MortyChoi
    MortyChoi @mortyccp ·

 

 

 

 

 

FYI, while the blog post certainly makes it sound like it would, this does not actually work with Go 1.12 and below. Apparently this was released without having been tested, but in fact that is a good thing because:

Recently this feature was merged into Go master, so starting from Go 1.13 this might really work.

Markdown and quick actions are supported
0 Assignees
None
Epic
None
Milestone
Backlog
Time tracking

No estimate or time spent
Due date
None
Labels
Accepting merge requests Create [DEPRECATED] GitLab.com Priority Manage [DEPRECATED] authentication backend customer devopscreate feature
Weight
None

Health status

None

Confidentiality
Not confidential
Lock issue
Unlocked
24 participants
user avatar
user avatar
user avatar
user avatar
user avatar
user avatar
user avatar
Notifications
Reference: gitlab-org/gitlab-foss#45055