Commit Graph

23 Commits

Author SHA1 Message Date
David Pursehouse a7d3fa3064 Explicitly specify charset in constructor of String
Change-Id: Ie9a9f917503019e7fa51ccbc11a5a3518b74434b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-09-26 08:31:13 +09:00
David Pursehouse 30c6c75421 Deprecate Constants.CHARSET in favor of StandardCharsets.UTF_8
Change-Id: I3b748620f067582afef20f144feebe40d0332be2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-09-04 10:53:10 +09:00
Han-Wen Nienhuys f3ec7cf3f0 Remove further unnecessary 'final' keywords
Remove it from

 * package private functions.

 * try blocks

 * for loops

this was done with the following python script:

$ cat f.py
import sys
import re
import os

def replaceFinal(m):
  return m.group(1) + "(" +  m.group(2).replace('final ', '') + ")"

methodDecl = re.compile(r"^([\t ]*[a-zA-Z_ ]+)\(([^)]*)\)")

def subst(fn):
  input = open(fn)
  os.rename(fn, fn + "~")

  dest = open(fn, 'w')
  for l in input:
    l = methodDecl.sub(replaceFinal, l)
    dest.write(l)
  dest.close()


for root, dirs, files in os.walk(".", topdown=False):
  for f in files:
    if not f.endswith('.java'):
      continue

    full = os.path.join(root, f)
    print full
    subst(full)

Change-Id: If533a75a417594fc893e7c669d2c1f0f6caeb7ca
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2018-05-18 17:59:45 +02:00
David Pursehouse f07b60239c Consistently use Constants.CHARSET rather than StandardCharsets.UTF_8
Change-Id: I6714fc3666e1bced22abba94ceb700477349586e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-03-11 22:46:52 +01:00
Matthias Sohn b7651da4e8 Remove deprecated LfsProtocolServlet#getLargeFileRepository
Use LfsProtocolServlet##getLargeFileRepository(LfsRequest, String,
String) instead.

Change-Id: If68d06f2e6edbf843dc9d3b4e3b99c33b0fab766
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-03-10 19:41:33 +01:00
David Pursehouse 60b43c03e2 Add ConfigConstants.CONFIG_SECTION_LFS
DirCacheCheckout has a warning about non-localised string "lfs". Other
classes use org.eclipse.jgit.lfs.lib.Constants but that is not visible
to DirCacheCheckout.

Add a new constant in ConfigConstants and use that in DirCacheCheckout.

Replace existing uses of org.eclipse.jgit.lfs.lib.Constants.LFS with
the new constant, except where it is referring to the folder name.

Change-Id: I0f21b951babff9a2e579d68c4de0c62ee4bc23d4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-03-05 10:53:28 +09:00
Markus Duft c0103bc59d LFS: Enable LFS support for the CLI, better error handling
Enable LFS support for the CLI by registering the according filters.

Errors during filter creation must be propagated up the call stack, as a
failure to create a filter should be treated as fatal if the filter is
required.

Change-Id: I3833757839bdda97cd01b6c21c1613d199e2692d
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
2018-03-03 11:44:01 +01:00
Markus Duft d3ed64bcd4 LFS: support merge/rebase/cherry-pick/diff/compare with LFS files
Respect merge=lfs and diff=lfs attributes where required to replace (in
memory) the content of LFS pointers with the actual blob content from
the LFS storage (and vice versa when staging/merging).

Does not implement general support for merge/diff attributes for any
other use case apart from LFS.

Change-Id: Ibad8875de1e0bee8fe3a1dffb1add93111534cae
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-03-03 11:39:43 +01:00
Markus Duft c0bb992845 LFS: pre-push upload support
If JGit built in LFS support is enabled for the current repository (or
user/system), any existing pre-push hook will cause an exception for the
time beeing, as only a single pre-push hook is supported.

Thus either native pre-push hooks OR JGit built-in LFS support may be
enabled currently, but not both.

Change-Id: Ie7d2b90e26e948d9cca3d05a7a19489488c75895
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-27 18:32:45 +01:00
Markus Duft 94bcde663c LFS: Add remote download to SmudgeFilter
Transfer data in chunks of 8k Transferring data byte per byte is slow,
running checkout with CleanFilter on a 2.9MB file takes 20 seconds.
Using a buffer of 8k shrinks this time to 70ms.

Also register the filter commands in a way that the native GIT LFS can
be used alongside with JGit.

Implements auto-discovery of LFS server URL when cloning from a Gerrit
LFS server.

Change-Id: I452a5aa177dcb346d92af08b27c2e35200f246fd
Also-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
2018-02-16 18:27:25 +01:00
David Pursehouse 5f94e44308 LfsServerText: Fix formatting of invalidPathInfo message
The message is formatted as:

  Invalid pathInfo '/abc' does not match '/{SHA-256}'

but should be:

  Invalid pathInfo: '/abc' does not match '/{SHA-256}'

(i.e. including a colon) to be consistent with other messages.

Change-Id: Ic38aa7d33dd02d7954b95c331a73919a90c69991
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-07 20:02:59 +09:00
David Pursehouse 899696ed08 InvalidLongObjectIdException: Fix formatting of message
The message is formatted as:

  Invalid id: : abcde...

but should be:

  Invalid id: abcde...

Change-Id: Ie15cacdcf2f168edaee262e6cf8061ebfe9d998d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-07 20:02:59 +09:00
David Pursehouse cc3cf93eee DownloadTest: Use ExpectedException instead of try..catch..fail
Change-Id: I5547e64de6cea3d2200887b504245c69692f00b5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-07 20:02:59 +09:00
David Pursehouse a7e20cc77f FileLfsServlet: Do not pretty print json response
Pretty printing the response is useful for human readers, but most
(if not all) of the time, the response will be read by programs.

Remove it to avoid the additional overhead of the formatting and
extra bytes in the response. Adjust the test accordingly.

Note that LfsProtocolServlet already doesn't use pretty printing,
so this change makes FileLfsServlet's behavior consistent. In fact,
both classes now have duplicate Gson handling; this will be cleaned
up in a separate change.

Change-Id: I113a23403f9222f16e2c0ddf39461398b721d064
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-07 08:42:13 +09:00
David Pursehouse 4940ea14b7 Add missing newlines at ends of Java files
Change-Id: Iead36f53d57ead0eb3edd3f9efb63b6630c9c20c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-07-25 10:37:21 +01:00
David Pursehouse b8c747abde LfsServerTest: Remove unnecessary suppression of 'restriction' warning
Change-Id: I23386808848746d201ca1ac13f114c06fc8e86c5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-09-22 09:46:26 +02:00
David Pursehouse 2fc1cebfc9 DownloadTest: Use enums from org.apache.http.HttpStatus
Instead of using hard-coded HTTP status codes, use the enums
which makes it a bit easier to see what's expected.

Change-Id: I2da5d25632f374b8625d64da4df70d1c9c406bb1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-08-20 00:54:52 +02:00
David Pursehouse f703a5eb2c Fix typo in email address in copyright headers
Change-Id: Ib7b73d7d37574682bb58f969822c842e4e579233
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-28 16:05:22 +09:00
David Pursehouse ddb12b003f LfsServerTest: Treat response body as UTF-8 when decoding error message
Change-Id: I495f0b60b7128fff27502641e6a5d05f888d4e8a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-28 12:17:45 +09:00
David Pursehouse e27eab26e2 FileLfsServlet: Include error message in response body
According to the specification [1], the error response body must
include the error message in json format.

[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md#response-errors

Change-Id: I79e7a841d230fdedefa53b9c6d2d477e81e1f9e6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-27 11:07:05 +09:00
David Pursehouse 2e652aebab FileLfsServlet: Return HTTP 422 instead of 400
According to the specification [1], the error response status code
should be 422 when there is a validation error with one or more of
the objects in the request

[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md#response-errors

Change-Id: Id03fe00a2109b896d9a154228a14a33bce5accc3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-27 11:01:14 +09:00
Saša Živkov b72fc2b494 Make the FileLfsRepository thread safe
The FileLfsRepository.out member could have been accessed from multiple
threads which would corrupt the content.

Don't store the AtomicObjectOutputStream in the FileLfsRepository.out but
move it to the ObjectUploadListener which is instantiated per-request.

Add a parallel upload test.

Change-Id: I62298630e99c46b500d376843ffcde934436215b
Signed-off-by: Saša Živkov <sasa.zivkov@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-03-22 17:26:46 +01:00
Matthias Sohn 3bae524f6f Support LFS protocol and a file system based LFS storage
Implement LfsProtocolServlet handling the "Git LFS v1 Batch API"
protocol [1]. Add a simple file system based LFS content store and the
debug-lfs-store command to simplify testing.

Introduce a LargeFileRepository interface to enable additional storage
implementation while reusing the same protocol implementation.

At the client side we have to configure the lfs.url, specify that
we use the batch API and we don't use authentication:

  [lfs]
	  url = http://host:port/lfs
	  batch = true
  [lfs "http://host:port/lfs"]
	  access = none

the git-lfs client appends the "objects/batch" to the lfs.url.

Hard code an Authorization header in the FileLfsRepository.getAction
because then git-lfs client will skip asking for credentials. It will
just forward the Authorization header from the response to the
download/upload request.

The FileLfsServlet supports file content storage for "Large File
Storage" (LFS) server as defined by the Github LFS API [2].

- upload and download of large files is probably network bound hence use
  an asynchronous servlet for good scalability
- simple object storage in file system with 2 level fan-out
- use LockFile to protect writing large objects against multiple
  concurrent uploads of the same object
- to prevent corrupt uploads the uploaded file is rejected if its hash
  doesn't match id given in URL

The debug-lfs-store command is used to run the LfsProtocolServlet and,
optionally, the FileLfsServlet which makes it easier to setup a
local test server.

[1]
https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md
[2] https://github.com/github/git-lfs/tree/master/docs/api

Bug: 472961
Change-Id: I7378da5575159d2195138d799704880c5c82d5f3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
2016-02-04 17:49:43 +01:00