Without this dependency running JGit CLI in Eclipse will hit
ClassNotFoundExceptions when HttpClient is trying to log something.
Change-Id: I2d50d9a18fac4c302de2c3a16c07f90ce3e5072e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Adds a JGit built-in implementation of the "git lfs clean" filter. This
filter should do the same as the one described in [1]. But since this
filter is written in Java and can be called by JGit without forking new
processes it should be much faster
[1]
https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-clean.1.ronn
Change-Id: If60e387e97870245b4bd765eda6717eb84cffb1d
Add a storage implementation storing large objects in Amazon S3.
The AmazonS3Repository pre-signs download and upload requests.
AWS access and secret key are expected to be in the
$HOME/.aws/credentials file in the following format:
[default]
accessKey = ...
secretKey = ...
Use AWS version 4 request signing [1] because it is more secure and
supported by all regions. The version 3 signing is not supported in
newer regions.
In follow up changes we should:
- implement getVerifyAction() and do actual verification. Subclasses of
S3Repository can implement caching for object meta data (size) in order
to avoid extra roundtrips to S3. Verification should ensure that meta
data store and content of S3 storage are in sync
- HEAD request used in S3Repository.getSize() seems to always return
Content-length 0 in contrast to the documentation [2]. So getSize() does
detect if the object exists in S3 or not but in case the object exists
it always returns size 0
[1] http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
[2] https://forums.aws.amazon.com/thread.jspa?threadID=223616
Change-Id: Ic47f094928a259e5264c92b3aacf6d90210907a8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Implement LfsProtocolServlet handling the "Git LFS v1 Batch API"
protocol [1]. Add a simple file system based LFS content store and the
debug-lfs-store command to simplify testing.
Introduce a LargeFileRepository interface to enable additional storage
implementation while reusing the same protocol implementation.
At the client side we have to configure the lfs.url, specify that
we use the batch API and we don't use authentication:
[lfs]
url = http://host:port/lfs
batch = true
[lfs "http://host:port/lfs"]
access = none
the git-lfs client appends the "objects/batch" to the lfs.url.
Hard code an Authorization header in the FileLfsRepository.getAction
because then git-lfs client will skip asking for credentials. It will
just forward the Authorization header from the response to the
download/upload request.
The FileLfsServlet supports file content storage for "Large File
Storage" (LFS) server as defined by the Github LFS API [2].
- upload and download of large files is probably network bound hence use
an asynchronous servlet for good scalability
- simple object storage in file system with 2 level fan-out
- use LockFile to protect writing large objects against multiple
concurrent uploads of the same object
- to prevent corrupt uploads the uploaded file is rejected if its hash
doesn't match id given in URL
The debug-lfs-store command is used to run the LfsProtocolServlet and,
optionally, the FileLfsServlet which makes it easier to setup a
local test server.
[1]
https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md
[2] https://github.com/github/git-lfs/tree/master/docs/api
Bug: 472961
Change-Id: I7378da5575159d2195138d799704880c5c82d5f3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Experimental flag to turn on the KetchLeader within this daemon JVM.
This is a manually elected leader process, set from the command line.
Remote followers for each repository are configured per-repository
using remote sections with ketch-type = FULL. For example:
Manually elected leader's $GIT_DIR/config:
[ketch]
name = A
[remote "A"]
ketch-type = FULL
[remote "B"]
url = git://127.0.0.1:9421/sample.git
ketch-type = FULL
[remote "C"]
url = git://127.0.0.1:9422/sample.git
ketch-type = FULL
Replica B and C daemons:
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9421 \
--base-path=$HOME/ketch_test/follower_one \
$HOME/ketch_test/follower_one &
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9422 \
--base-path=$HOME/ketch_test/follower_two \
$HOME/ketch_test/follower_two &
Change-Id: I165f85970a77e16b5263115290d685d8a00566f5
This tool scans all references in the repository and writes out a new
reference pointing to a single commit whose root tree is a RefTree
containing the current refs of this repository.
It alway skips storing the reference it will write to, avoiding the
obvious cycle.
Change-Id: I20b1eeb81c55dc49dd600eac3bf8f90297394113
Pushing with JGit commandline to e.g. Github failed with "unauthorized"
since HttpUrlConnection calls the configured authenticator implicitly.
The problem is that during a push two requests are sent to the server,
first a GET and then a POST (containing the pack data). The first GET
request sent anonymously is rejected with 401 (unauthorized). When an
Authenticator is installed the java.net classes will use the
Authenticator to ask the user for credentials and retry the request.
But this happens under the hood and JGit level code doesn't see that
this happens.
The next request is the POST but since JGit thinks the first GET request
went through anonymously it doesn't add authentication headers to the
POST request. This POST of course also fails with 401 but since this
request contains a lot of body-data streamed from JGit (the pack file!)
the java.net classes can't simply retry the request with authorization
headers. The whole process fails.
Fix this by using Apache httpclient which doesn't use Authenticator to
retrieve credentials. Instead initialize TransportCommand to use the
default credential provider if no other credentials provider was set
explicitly. org.eclipse.jgit.pgm.Main sets this default for the JGit
command line client.
Change-Id: Ic4e0f8b60d4bd6e69d91eae0c7e1b44cdf851b00
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Update the project-specific Eclipse settings to replace the use of the
org.eclipse.jdt.annotation.Nullable class the new JGit-specific
@Nullable annotation. I verified that Eclipse reports errors when the
return value of a method annotated with
@org.eclipse.jgit.annotations.Nullable is dereferenced without a null
check.
Also remove the Maven and MANIFEST.MF dependencies on
org.eclipse.jdt.annotation.
Eclipse null analysis uses three annotations: @Nullable, @NonNull and
@NonNullByDefault. All three are updated in this patch because it is
invalid to set the Eclipse preferences to empty values. So far only
@Nullable has been introduced in org.eclipse.jgit.annotations.
My personal preference is to follow the advice in Effective Java and
avoid the null-return idiom, and to avoid passing null values in
general. This sets the expectation is that arguments and return types
are assumed non-null unless otherwise documented. If that is the
expectation, then consistent application of @NonNull is redundant and
hurts readability by cluttering the code, obscuring the occasional
@Nullable annotation that really requires attention.
If the JGit community decides there is value in using the @NonNull and
@NonNullByDefault annotations we can add them--this change configures
Eclipse to use them.
Change-Id: I9af1b786d1b44b9b0d9c609480dc842df79bf698
Signed-off-by: Terry Parker <tparker@google.com>
This header was removed unintentionally from some bundles in
3a4a5a4e57. Restore it to ensure lazy
activation of bundles.
Change-Id: I1f841f978fb93278e3ec0533a01f1363510dd976
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In Bug 476164 it was reported that EGit doesn't start when the platform
comes with jsch 0.1.51 while this version of EGit/JGit brings jsch
0.1.53. This could be caused by outdated uses-clauses. Hence recompute
them using PDE tooling.
Bug: 476164
Change-Id: I185ba097884ead9cd034eba842bd3bf34181a99b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
A special options handler is added to properly handle the short -u alias
of the option.
The "normal" mode is not supported by this patch, because this mode of
listing untracked files is not
supported by the org.eclipse.jgit.lib.IndexDiff class. This mode is not
necessary for my use case. It can be added later if anyone really needs
it.
The StatusTest is updated to cover all possible combinations of the
--porcelain and --untracked-files options.
Bug: 459319
Change-Id: I305ac95739cfed0c16735e0987844e57fa27e236
Signed-off-by: Kaloyan Raev <kaloyan.r@zend.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Since we updated minimum Java version to Java 7 the console bundle
doesn't need to be a separate bundle anymore. Move the contained classes
to the pgm bundle which is using these classes.
Change-Id: If8e6f2d7405fdfe6f4b178673b4ccf99c67d4b64
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
- don't mark them as singleton to allow coexistence of multiple versions
in the same installation
- add missing version qualifier to Eclipse-SourceBundle header
see
https://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg10524.html
Change-Id: Ie4e028038f5a1d3e18b0be06c3d2ea82e7f9068d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
All bugs reported for gc have been fixed and it seems we reached a
stable implementation.
Change-Id: I78a96ee2103beb48325da0f6ee10b2498bdc0267
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Allow use of ArchiveCommand without depending on the jgit command-line
tools.
To avoid complicating the process of installing and upgrading JGit,
this does not add a dependency by the org.eclipse.jgit bundle on
commons-compress. Instead, the caller is responsible for registering
any formats they want to use by calling ArchiveCommand.registerFormat.
This patch puts functionality that requires an archiver into a
separate org.eclipse.jgit.archive bundle for people who want it. One
can use it by calling ArchiveCommand.registerFormat directly to
register its formats or by relying on OSGi class loading to load
org.eclipse.jgit.archive.FormatActivator, which takes care of
registration automatically.
Once the appropriate formats are registered, you can make a tar or zip
from a git tree object as follows:
ArchiveCommand cmd = git.archive();
try {
cmd.setTree(tree).setFormat(fmt).setOutputStream(out).call();
} finally {
cmd.release();
}
Change-Id: I418e7e7d76422dc6f010d0b3b624d7bec3b20c6e
Provide static registerFormat and unregisterFormat methods to allow
formats to register themselves without the ArchiveCommand code being
aware of them.
Register the basic "zip" and "tar" support at bundle activation time
(and deregister them when unloading the bundle). For anyone using
this code as an OSGi plugin it should continue to just work.
The jgit program does not load org.eclipse.jgit.pgm as an OSGi bundle,
so let the Archive command register the formats it uses explicitly
with registerFormat.
Change-Id: Id39c03ea6923d0aed8316ed7b6bd04d5ced570a7
Translatable texts aren't API and shouldn't require maintenance of
@since tags to prevent API warnings.
Change-Id: I228ff37f17c0e792a6bc188c463a0d19138e88ac
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This breaks all existing callers once. Applications are not supposed
to build against the internal storage API unless they can accept API
churn and make necessary updates as versions change.
Change-Id: I2ab1327c202ef2003565e1b0770a583970e432e9
It stopped working when we moved to the Eclipse foundation's Gerrit
server since it doesn't use the Gerrit internal user store but LDAP.
Instead, since 2.0, we use the Eclipse foundation's automatic IP log
generator [1] to generate IP logs for releasing jgit and egit.
[1] http://www.eclipse.org/projects/ip_log_selector.php
Change-Id: I98dc65efb62909bc0258e6c680df0c93a57e9677
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Unlike ZIP files, tar files do not treat symlinks as ordinary files
with a different mode, so tar support involves a little more code than
would be ideal.
Change-Id: Ica2568f4a0e443bf4b955ef0c029bc8eec62d369
The Apache Commons Compress library provides a similar interface to
java.util.zip with some features not found in java.util.zip, including
support for inclusion of metadata (file mode and symlink targets) and
support for multiple file formats (zip, .tar.xz, etc).
Use it, in preparation for making use of these features. No
functional change intended yet.
A previous version of this patch used plexus-archiver. That is a
heavier-weight dependency and offers a less convenient interface.
Thanks to James Moger and Chris Aniszczyk for advice.
Change-Id: Id01146950bb9c18dae0169311e3cde2c3bfa675e
C Git's "git archive" command represents a tree object using a
standard archival format like tar, zip, or tgz, ready for consumption
by other, git-unaware users or tools.
Add a bare-bones analagous "jgit archive" command to show what is
possible, supporting only ZIP format for now. It uses java.util.zip
which is not aware of the InfoZIP extensions for representing symlinks
and file permissions, so symlinks, executable files, and submodule
entries are represented as plain text files.
Making this functionality available from the library, improving
handling of special entries, and support for other output formats are
left for later patches. Ultimately the intent is to offer a
TreeArchiveStream class for use by web frontends like Gitiles to offer
"download as zip/tgz/txz" links and use by, for example, code search
tools to get easy access to the content of git tree objects.
Test with "jgit archive my-favorite-tree >out.zip".
Change-Id: Ib590f173ceff3df4b58493cecccd6b9a1b355e3d
The underlying problem is that System.out is a PrintWriter and
as such it does not throw exceptions on error, but rather just
sets a flag and continues.
This changes replaces the use of System.out with a PrintWriter-like
writer that does not catch error, but instead throw them to the
caller.
Bug: 366243
Change-Id: I44405edc4416e943b87f09a0f6ed041c6c51b046
This is a first basic implementation that displays current branch and
list of files of various status, but isn't as refined as its native
counterpart (e.g. does not say if we're ahead or behind the remote).
It's been helpful in the diagnostic of bug #347885.
Bug: 348318
CQ: 6769
Change-Id: Ifc35da608fbba652524c1b5b522e3c0d5369ad5e
Signed-off-by: François Rey <eclipse.org@francois.rey.name>
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Implements a garbage collector for FileRepositories. Main ideas are
copied from the garbage collector for DFS based repos
(DfsGarbageCollector). Added functionalities are
- pruning loose objects
- handling of the index
- packing refs
- handling of reflogs (objects referenced from reflog will not be
pruned/)
These are features of a GC which are not handled in this change and
which should come with subsequent changes:
- unpacking packed objects into loose objects (to support that pruning
packed objects doesn't delete them until they are older than two weeks)
- expiration of reflogs
- support for configuration parameters (e.g. gc.pruneExpire)
Change-Id: I14ea5cb7e0fd1b5c50b994fd77f4e05bfbb9d911
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Matthias Sohn