This provides a place to declare visibility restrictions and
transitive dependencies for each library.
Other targets should only declare dependencies on what they directly
use, making dependencies easier to maintain.
Trim the dependencies of org.eclipse.jgit:jgit to follow that rule.
It declares dependencies on Apache httpcomponents and the servlet
API but doesn't use them.
Tested:
* 'bazel build //...' succeeds
* applying the change https://gerrit-review.googlesource.com/90843
to a copy of Gerrit, following the instructions there, and running
'bazel test //...' in that copy of Gerrit still succeeds
Change-Id: I3ab958ce8b3227019cdbe4cc81e0f042e1541034
Breaking API for API implementors (not clients) in a minor release is ok
following OSGi semantic versioning rules.
Change-Id: I14bb4a3084b237b4cd893a130e148148cd1e5df9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This allows implementations to reject operations that do not
include proper authentication.
Change-Id: If301476d8fb56a0899e424be3789c7576097d185
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Since the introduction of generic type parameter inference in Java 7,
it's not necessary to explicitly specify the type of generic parameters.
Enable the warning in Eclipse, and fix all occurrences.
Change-Id: I9158caf1beca5e4980b6240ac401f3868520aad0
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Set missingOverrideAnnotation=warning in Eclipse compiler preferences
which enables the warning:
The method <method> of type <type> should be tagged with @Override
since it actually overrides a superclass method
Justification for this warning is described in:
http://stackoverflow.com/a/94411/381622
Enabling this causes in excess of 1000 warnings across the entire
code-base. They are very easy to fix automatically with Eclipse's
"Quick Fix" tool.
Fix all of them except 2 which cause compilation failure when the
project is built with mvn; add TODO comments on those for further
investigation.
Change-Id: I5772061041fd361fe93137fd8b0ad356e748a29c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Otherwise these methods may produce unexpected results if used for
strings that are intended to be interpreted locale independently.
Examples are programming language identifiers, protocol keys, and HTML
tags. For instance, "TITLE".toLowerCase() in a Turkish locale returns
"t\u0131tle", where '\u0131' is the LATIN SMALL LETTER DOTLESS I
character.
See
https://docs.oracle.com/javase/8/docs/api/java/lang/String.html#toLowerCase--http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html
Bug: 511238
Change-Id: Id8d8f37d84d62239c918b81f8d883ed798d87656
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Test plan:
$ bazel build all
$ unzip -t bazel-genfiles/all.zip
Archive: bazel-genfiles/all.zip
testing: libhttp-apache.jar OK
testing: libjgit-archive.jar OK
testing: libjgit-lfs-server.jar OK
testing: libjgit-lfs.jar OK
testing: libjgit-servlet.jar OK
testing: libjgit.jar OK
testing: libjunit.jar OK
No errors detected in compressed data of bazel-genfiles/all.zip.
Change-Id: I9e6c60898ccc6d2a4557ec7544c297442a9702b4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The same was already done for ObjectUploadListener in I5b0fb1220.
Change-Id: Ie8a79f715fe69bed9331962fb478f7e35acf8680
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Server implementations may need to check what type of operation is
being performed. Add helper methods to make it a bit simpler.
Change-Id: Ia33ed6699ebcb9000ef514ce79bcddbebf94e1c5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
So that they can be used either by third party LFS server implementations
or from within other parts of JGit's internal LFS implementation.
Change-Id: I58da6230247204588e017c010ce5b14e4615448d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Externalize the error message and make it more specific. Also emit
an error log.
Change-Id: Ie7b1c90c54673bfb8e133fb0aa19a117d4ca6587
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add a new exception type that server implementations can throw when a
client attempts to make an unauthorized LFS operation, which will result
in HTTP 401 Unauthorized being returned to the client.
An example of this is a Gerrit server that rejects a request to perform
an LFS operation on a ref that is not visible to the caller.
As defined in the LFS spec [1] the request may include authentication,
and per RFC 2616 [2], "401 response indicates that authorization has been
refused for those credentials".
[1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
[2] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
Change-Id: I2aa22e2144df5fb7972df0e3285b77b08ecc63f2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Providing own implementation to doGet/doPut methods is troublesome when
this method is private.
Change-Id: I098cdc5cb90410eaaebc56c88c2d9e168584dd6d
Signed-off-by: Jacek Centkowski <geminica.programs@gmail.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.5:
Use the same variable to check and extract LFS object id
Change-Id: I314bd4373f40843c68853b3999f60d85e08628d9
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
It is easier to maintain when the same variable is used for both check
and extraction of LFS object id.
Change-Id: I5406f9bc4a085aa164c4565a9667ad2925105190
Signed-off-by: Jacek Centkowski <geminica.programs@gmail.com>
The class AtomicObjectOutputStream should be available to all lfs
related classes, not only to the server side. Move the class from
org.eclipse.jgit.lfs.server.fs to org.eclipse.jgit.lfs.internal to
achieve that.
Change-Id: I028e1c9ec7c21f316340b21d558b9a6b77e2060d
Guide implementors which exception to throw in case of errors.
Change-Id: I74fb76cdf6b7cdef513f3fe8c144572e869cc533
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Instead of returning null, LfsProtocolServlet#getLargeFileRepository
should throw LfsUnavailable.
If null is returned, throw a generic LfsException.
Handle LfsException as an internal server error and return HTTP 500.
Change-Id: I33e2a19fcc0fde8aaf0f703860c8fa8ce2de2db5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Since [1], the git-lfs specification allows the server to return
HTTP 507 if there is insufficient storage for the uploaded object(s).
Add a new exception class, which implementations may throw from the
getRepository() method, causing HTTP 507 to be returned to the client.
[1] https://github.com/github/git-lfs/pull/1473
Change-Id: If5bc0a35fcf870d4216af6ca2f7c8924689ef9c5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The git-lfs specification [1] describes the following optional status codes
that may be returned:
429 - The user has hit a rate limit with the server. Though the API does
not specify any rate limits, implementors are encouraged to set some
for availability reasons.
509 - Returned if the bandwidth limit for the user or repository has been
exceeded. The API does not specify any bandwidth limit, but implementors
may track usage.
Add two new exception classes to represent these cases. Implementations may
throw these from #getLargeFileRepository(), causing the corresponding HTTP
status codes to be returned to the client.
[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md
Change-Id: I7b93f3cf90f7344c90b1587e07927fdeb167097e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
If the message is not sent, the client shows:
Unable to parse HTTP response for POST http://admin@localhost:8080/test-project/info/lfs/objects/batch
Change-Id: I8b72d1aded2bcd41b7389676e2373034625a1379
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
If the Content-Type is not set on error responses, the git-lfs client
does not read the body which contains the error message, and instead
just displays a generic error message.
Also set the charset on the Content-Type header.
Change-Id: I88e6f07f20b622a670e7c5063145dffb8b630aee
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Pretty printing is only used for outputting json content, which is
interpreted by the client and does not need to be pretty printed.
Change-Id: I48e0280241b6b0f5706300ae0f4c9bc461a89110
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Trying to open a new writer on the response causes an illegal state
exception and the response is not sent.
Change-Id: Ic718d23cfb3e74f5691cc2aea7283003af7df207
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
bb9988c2 changed the signature of getLargeFileRepository() which is only
breaking implementors which is ok according to OSGi semantic versioning
rules.
Change-Id: I68bda7900b72e217571f74aee53705167f8100a2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Classes implementing the LFS servlet should be able to inspect the
objects given in the request.
Add a getObjects method. Make the LfsObject class public, and add
accessor methods.
Change-Id: I27961679f620eb3a89dc8521aadd4ea2f936c60e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
According to the specification [1] the server may return the following
HTTP error responses:
- 403: The user has read, but not write access.
- 404: The repository does not exist for the user.
- 422: Validation error with one or more of the objects in the request.
In the current implementation, however, getLargeFileRepository can only
return null to indicate an error. This results in the error code:
- 503: Service Unavailable
being returned to the client regardless of what the actual reason was.
Add exception classes to cover these cases, derived from a common base
exception, and change the specification of getLargeFileRepository to throw
the base exception.
In LfsProtocolServlet#post, handle the new exceptions and send back the
appropriate HTTP responses as mentioned above.
The specification also mentions several other optional response codes (406,
429, 501, and 509) but these are not implemented in this commit. It should
be trivial to implement them in follow-up commits.
[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md#response-errors
Change-Id: I91be6165bcaf856d0cefc533882330962e2fc9b2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Passing the request and path to the method will allow implementations
to have more control over determination of the backend, for example:
- return different backends for different requests
- accept or refuse requests based on request characteristics
- etc
Change-Id: I1ec6ec54c91a5f0601b620ed18846eb4a3f46783
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
According to the specification [1], the error response status code
should be 422 when there is a validation error with one or more of
the objects in the request
[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md#response-errors
Change-Id: Id03fe00a2109b896d9a154228a14a33bce5accc3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The location of the API v1 documentation has changed. Update the
links accordingly.
Change-Id: If0148a0b573c474bbe157fcb7e6674c0055fe8b4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
We ignored the returned concatenation of host name and port number. Fix
this and use a StringBuilder to avoid creation of unnecessary String
objects.
Change-Id: I61fac639d4a4c95412eb41a0f9131d0c38aca794
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The FileLfsRepository.out member could have been accessed from multiple
threads which would corrupt the content.
Don't store the AtomicObjectOutputStream in the FileLfsRepository.out but
move it to the ObjectUploadListener which is instantiated per-request.
Add a parallel upload test.
Change-Id: I62298630e99c46b500d376843ffcde934436215b
Signed-off-by: Saša Živkov <sasa.zivkov@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add a storage implementation storing large objects in Amazon S3.
The AmazonS3Repository pre-signs download and upload requests.
AWS access and secret key are expected to be in the
$HOME/.aws/credentials file in the following format:
[default]
accessKey = ...
secretKey = ...
Use AWS version 4 request signing [1] because it is more secure and
supported by all regions. The version 3 signing is not supported in
newer regions.
In follow up changes we should:
- implement getVerifyAction() and do actual verification. Subclasses of
S3Repository can implement caching for object meta data (size) in order
to avoid extra roundtrips to S3. Verification should ensure that meta
data store and content of S3 storage are in sync
- HEAD request used in S3Repository.getSize() seems to always return
Content-length 0 in contrast to the documentation [2]. So getSize() does
detect if the object exists in S3 or not but in case the object exists
it always returns size 0
[1] http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
[2] https://forums.aws.amazon.com/thread.jspa?threadID=223616
Change-Id: Ic47f094928a259e5264c92b3aacf6d90210907a8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Implement LfsProtocolServlet handling the "Git LFS v1 Batch API"
protocol [1]. Add a simple file system based LFS content store and the
debug-lfs-store command to simplify testing.
Introduce a LargeFileRepository interface to enable additional storage
implementation while reusing the same protocol implementation.
At the client side we have to configure the lfs.url, specify that
we use the batch API and we don't use authentication:
[lfs]
url = http://host:port/lfs
batch = true
[lfs "http://host:port/lfs"]
access = none
the git-lfs client appends the "objects/batch" to the lfs.url.
Hard code an Authorization header in the FileLfsRepository.getAction
because then git-lfs client will skip asking for credentials. It will
just forward the Authorization header from the response to the
download/upload request.
The FileLfsServlet supports file content storage for "Large File
Storage" (LFS) server as defined by the Github LFS API [2].
- upload and download of large files is probably network bound hence use
an asynchronous servlet for good scalability
- simple object storage in file system with 2 level fan-out
- use LockFile to protect writing large objects against multiple
concurrent uploads of the same object
- to prevent corrupt uploads the uploaded file is rejected if its hash
doesn't match id given in URL
The debug-lfs-store command is used to run the LfsProtocolServlet and,
optionally, the FileLfsServlet which makes it easier to setup a
local test server.
[1]
https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md
[2] https://github.com/github/git-lfs/tree/master/docs/api
Bug: 472961
Change-Id: I7378da5575159d2195138d799704880c5c82d5f3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>