* master:
Revert "Prepend hostname to subsection used to store file timestamp resolution"
Remove use of org.bouncycastle.util.encoders.Hex
Remove use of org.bouncycastle.util.io.TeeOutputStream
Make the IMatcher public API
SimilarityRenameDetector: Fix inconsistent indentation
Use indexOf(char) and lastIndexOf(char) rather than String versions
Reorder modifiers to follow Java Language Specification
GitmoduleEntry: Remove redundant import of class from same package
Remove redundant "static" qualifier from enum declarations
Change-Id: Ie5a3f55229307c900c8974fcaeb1e77c1c87e87f
Purpose of the next branch is preparation of breaking changes before
we know when we will release it. Bump the version to the next major
release 6.0.0-SNAPSHOT in order to prevent build results compete with
build results from the version currently used on the master branch
(5.7.0-SNAPSHOT).
Bug: 560424
Change-Id: Iba64a0124f5bfc167098d829a7ef29650948e2d0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.6:
Add ability to redirect stderr from git hooks
Add possibility to get pure stderr output from AbortedByHookException
Change-Id: Ifc02675542dad6ced25fdd8b9fae80b5736db688
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This will change the behavior in the CLI to resemble that of C-Git more
closely by printing the stderr of the hooks to the CLI stderr
independently of the exit code of the hook.
This is also useful for the corresponding EGIT-Change, which will add
the ability to show the hook output in eclipse.
With this also the stderr can be shown even if the exit code is 0.
Bug: 553471
Change-Id: Ie7bc503fe39e270e9b93dd1108b5879f02a12b4c
Signed-off-by: Tim Neumann <Tim.Neumann@advantest.com>
Instead of just looking for a substring match of user.signingKey
in a key's user ID implement the GPG matching formats[1] for:
'=' Full exact match
'<' Full exact match of the e-mail address
'@' Substring match within the e-mail address only
'*' General case-insensitive substring match (default)
When user.signingKey is not set, the committer's e-mail address is
used by default. In that case, use '<', i.e., require an exact match
on the OpenPGP e-mail address.
Also handle the optional "0x" prefix for (partial) key fingerprints.
[1] https://www.gnupg.org/documentation/manuals/gnupg/Specify-a-User-ID.html
Bug: 550335
Change-Id: I6ce482a099ff1a0dc9de45435cd4d3ec5b504f12
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The placeholders in manifest and plugin.properties did not match. To
avoid similar issues, all placeholders have been changed to
Bundle-Vendor and Bundle-Name now.
Bug:548503
Change-Id: Ibd4b9bc237b323e614506b97e5fbc99416365040
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
The git config entries "http.cookieFile" and
"http.saveCookies" are correctly evaluated.
Bug: 488572
Change-Id: Icfeeea95e1a5bac3fa4438849d4ac2306d7d5562
Signed-off-by: Konrad Windszus <konrad_w@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Also now use JcaKeyBoxBuilder constructor in
BouncyCastleGpgKeyLocator.readKeyBoxFile(Path).
CQ: 19868
CQ: 19869
CQ: 19870
Change-Id: I45bd80e158aecd90448b0c7e59615db27aaef892
Signed-off-by: Brandon Weeks <bweeks@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This also includes a change to generating the jgit CLI jar. Shading is
no longer possible because it breaks the signature of BouncyCastle.
Instead, the Spring Boot Loader Maven plug-in is now used to generate an
executable jar.
Bug: 382212
Change-Id: I35ee3d4b06d9d479475ab2e51b29bed49661bbdc
Also-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Medha Bhargav Prabhala <mprabhala@salesforce.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
and allow package org.eclipse.jgit.http.server to use package
org.eclipse.jgit.internal.transport.parser.
Change-Id: Ief330c3e75a735853d0a5a265a9ff56fb5128b99
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This simplifies the BaseReceivePack class and decreases its API
surface, which should make merging with ReceivePack easier.
Inspired by 6aca8899a5 (Move first line
parsing for v0/v1 pack negotiation out of UploadPack, 2018-09-17).
Change-Id: I1fc175d15aa7cb5968c26fc83a95480403af617c
Add a new ssh client implementation based on Apach MINA sshd 2.0.0.
This implementation uses JGit's own config file parser and host entry
resolver. Code inspection of the Apache MINA implementation revealed
a few bugs or idiosyncrasies that immediately would re-introduce bugs
already fixed in the past in JGit.
Apache MINA sshd is not without quirks either, and I had to configure
and override more than I had expected. But at least it was all doable
in clean ways.
Apache MINA boasts support for Bouncy Castle, so in theory this should
open the way to using more ssh key algorithms, such as ed25519.
The implementation is in a separate bundle and is still not used in
the core org.eclipse.jgit bundle. The tests re-use the ssh tests from
the core test bundle.
Bug: 520927
Change-Id: Ib35e73c35799140fe050d1ff4fb18d0d3596580e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Move the bulk of the basic parsing and host entry handling into a
new class OpenSshConfigFile that has no dependencies on any concrete
ssh implementation. Make the existing OpenSshConfig use the new
parser.
Introduce a new class SshConstants collecting all the various ssh-
related string literals. Also use TreeMaps with a case-insensitive
key comparator instead of converting keys to uppercase. Add a test
to verify that keys are matched case-insensitively.
Most of the parsing code was simply moved, except that the new
parser supports looking up entries given host name, port, and user
name, and can thus handle more %-substitutions correctly. This
feature is not yet used and cannot be used with JSch since JSch
only has a ConfigRepository.getConfig(String) interface.
The split is still worth the trouble as it opens the way to using
another ssh client altogether. Apache MINA sshd, for instance,
resolves host entries giving host name, port, and user name.
(Apache MINA has a built-in ssh config handling, but that has
problems, too: its pattern matching is case-insensitive, and its
merging of host entries if several match is not the same as in
OpenSsh. But with this refactoring, it will be possible to plug in
OpenSshConfigFile into an Apache MINA sshd client without dragging
along JSch.)
One test case that doesn't make sense anymore has been removed. It
tested that repeatedly querying for a host entry returned the same
object. That is no longer true since the caching has been moved to
a deeper level.
Bug: 520927
Change-Id: I6381d52b29099595e6eaf8b05c786aeeaefbf9cc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Do not export o.e.j.internal.transport.parser as public package;
restrict visibility to org.eclipse.jgit.test only.
Add two packages that were not listed at all (o.e.j.internal.revwalk
and o.e.j.internal.submodule) marked as x-internal:=true.
Change-Id: I9188356075515ad354b724102fbd6304b682de6a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
In protocol v0/v1 pack negotiation, the first want line contains the
options the client wants in effect. This parsing is done in UploadPack
but it doesn't have any interaction with that class.
Move the code to its own class and package, mark the current one
as deprecated (it is public API) and add unit tests.
Take the chance to move the parsing code from the constructor to a
factory method, making the class a simple container of results.
Change-Id: I1757f535dda78a4111a1c12c3a3b455a4b6f0c51
Signed-off-by: Ivan Frade <ifrade@google.com>
* stable-4.11:
Remove package import for javax.servlet.http from org.eclipse.jgit
Add missing @since tag and silence API error
Change-Id: I2783a15ead26ab19de31a8fb3bfb148ef19de91a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Transfer data in chunks of 8k Transferring data byte per byte is slow,
running checkout with CleanFilter on a 2.9MB file takes 20 seconds.
Using a buffer of 8k shrinks this time to 70ms.
Also register the filter commands in a way that the native GIT LFS can
be used alongside with JGit.
Implements auto-discovery of LFS server URL when cloning from a Gerrit
LFS server.
Change-Id: I452a5aa177dcb346d92af08b27c2e35200f246fd
Also-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
This is a simple writer to create reftable formatted files. Follow-up
commits will add support for reading from reftable, debugging
utilities, and tests.
Change-Id: I3d520c3515c580144490b0b45433ea175a3e6e11
JGit already had some fsck-like classes like ObjectChecker which can
check for an individual object.
The read-only FsckPackParser which will parse all objects within a pack
file and check it with ObjectChecker. It will also check the pack index
file against the object information from the pack parser.
Change-Id: Ifd8e0d28eb68ff0b8edd2b51b2fa3a50a544c855
Signed-off-by: Zhen Chen <czhen@google.com>
* stable-4.8:
Use a dedicated executor to run auto-gc in command line interface
Allow to use an external ExecutorService for background auto-gc
Fetch: Add --recurse-submodules and --no-recurse-submodules options
Fix capitalization of command help summaries
Change-Id: I7c85f11daa34c11c7f6389de885a2183a686197e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
If set use the external executor, otherwise use JGit's own simple
WorkQueue. Move WorkQueue to an internal package so we can reuse it
without exposing it in the public API.
Change-Id: I060d62ffd6692362a88b4bf13ee07b0dc857abe9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This implementation is derived straight from the description written
in RFC 3174. On Mac OS X with Java 1.8.0_91 it offers similar
throughput as MessageDigest SHA-1:
system 239.75 MiB/s
system 244.71 MiB/s
system 245.00 MiB/s
system 244.92 MiB/s
sha1 234.08 MiB/s
sha1 244.50 MiB/s
sha1 242.99 MiB/s
sha1 241.73 MiB/s
This is the fastest implementation I could come up with. Common SHA-1
implementation tricks such as unrolling loops creates a method too
large for the JIT to effectively optimize, resulting in lower overall
hashing throughput. Using a preprocessor to perform the register
renaming of A-E also didn't help, as again the method was too large
for the JIT to effectively optimize.
Fortunately the fastest version is a naive, straight-forward
implementation very close to the description in RFC 3174.
Change-Id: I228b05c4a294ca2ad51386cf0e47978c68e1aa42
Add new variation of TreeFilter in order to detect LFS pointer files in
the repository.
Additionally, update LfsPointer to support the legacy version URL [1] as
described in [2], and to allow arbitrary fields in the pointer file.
[1] https://hawser.github.com/spec/v1
[2] https://github.com/git-lfs/git-lfs/blob/master/docs/spec.md
Change-Id: I621eb058619fb1b78888a54c4b60bb110a722fc3
Signed-off-by: Dariusz Luksza <dariusz@luksza.org>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
MonotonicClock can be implemented to provide more certainity about
time than the standard System.currentTimeMillis() can provide. This
can be used by classes such as PersonIdent and Ketch to rely on
more certainity about time moving in a strictly ascending order.
Gerrit Code Review can also leverage this interface through its
embedding of JGit and use MonotonicClock and ProposedTimestamp to
provide stronger assurance that NoteDb time is moving forward.
Change-Id: I1a3cbd49a39b150a0d49b36d572da113ca83a786
Use Oxygen M3 Orbit repository which provides the bundles built using
the new orbit-recipe based build.
CQ: 11658
Change-Id: I7f3dcc966732b32830c75d5daa55383bd028d182
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The class AtomicObjectOutputStream should be available to all lfs
related classes, not only to the server side. Move the class from
org.eclipse.jgit.lfs.server.fs to org.eclipse.jgit.lfs.internal to
achieve that.
Change-Id: I028e1c9ec7c21f316340b21d558b9a6b77e2060d
The exception can be thrown in a various reason, and sometimes 403
Forbidden is not appropriate. Make the HTTP status code customizable.
Change-Id: If2ef6f454f7479158a4e28a12909837db483521c
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Now if refs are unreadable when serving an upload pack the handler
will fail due to the actual underlying failure. Previously all wants
would be rejected as invalid because Repository.getAllRefs() returned
an empty map.
Testing this required a new subclass of InMemoryRepository so that
an IOException could be injected at the correct time.
Signed-off-by: Michael Edgar <adgar@google.com>
Change-Id: Iac708b1db9d0ccce08c4ef5ace599ea0b57afdc0
Implement LfsProtocolServlet handling the "Git LFS v1 Batch API"
protocol [1]. Add a simple file system based LFS content store and the
debug-lfs-store command to simplify testing.
Introduce a LargeFileRepository interface to enable additional storage
implementation while reusing the same protocol implementation.
At the client side we have to configure the lfs.url, specify that
we use the batch API and we don't use authentication:
[lfs]
url = http://host:port/lfs
batch = true
[lfs "http://host:port/lfs"]
access = none
the git-lfs client appends the "objects/batch" to the lfs.url.
Hard code an Authorization header in the FileLfsRepository.getAction
because then git-lfs client will skip asking for credentials. It will
just forward the Authorization header from the response to the
download/upload request.
The FileLfsServlet supports file content storage for "Large File
Storage" (LFS) server as defined by the Github LFS API [2].
- upload and download of large files is probably network bound hence use
an asynchronous servlet for good scalability
- simple object storage in file system with 2 level fan-out
- use LockFile to protect writing large objects against multiple
concurrent uploads of the same object
- to prevent corrupt uploads the uploaded file is rejected if its hash
doesn't match id given in URL
The debug-lfs-store command is used to run the LfsProtocolServlet and,
optionally, the FileLfsServlet which makes it easier to setup a
local test server.
[1]
https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md
[2] https://github.com/github/git-lfs/tree/master/docs/api
Bug: 472961
Change-Id: I7378da5575159d2195138d799704880c5c82d5f3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Git Ketch is a multi-master Git repository management system. Writes
are successful only if a majority of participant servers agree. Acked
writes are durable against server failures as a majority of the
participants store all required objects.
Git Ketch is modeled on the Raft Consensus Algorithm[1]. A ketch
sailing vessel is faster and more nimble than a raft. It can also
carry more source codes.
Git Ketch front-loads replication costs, which vaguely resembles a
ketch sailing vessel's distinguishing feature of the main mast on the
front of the ship.
[1] https://raft.github.io/
Change-Id: Ib378dab068961fc7de624cd96030266660b64fb4
A group of updates can be applied by updating the tree in one step,
writing out a new root tree, and storing its SHA-1. If references
are stored in RefTrees, comparing two repositories is a matter of
checking if two SHA-1s are identical. Without RefTrees comparing two
repositories requires listing all references and comparing the sets.
Track the "refs/" directory as a root tree by storing references
that point directly at an object as a GITLINK entry in the tree.
For example "refs/heads/master" is written as "heads/master".
Annotated tags also store their peeled value with ^{} suffix, using
"tags/v1.0" and "tags/v1.0^{}" GITLINK entries.
Symbolic references are written as SYMLINK entries with the blob of
the symlink carrying the name of the symbolic reference target.
HEAD is outside of "refs/" namespace so it is stored as a special
"..HEAD" entry. This name is chosen because ".." is not valid in
a reference name and it almost looks like "../HEAD" which names
HEAD if the reader was inside of the "refs/" directory.
A new Command type is required to handle symbolic references and
peeled references.
Change-Id: Id47e5d4d32149a9e500854147edd7d93c1041a39
This will execute git commands (with arguments) specified on the command
line, handy for developing/debugging a sequence of arbitrary git
commands working on same repository.
The git working dir path can be specified via Java system property
"git_work_tree". If not specified, current directory will be used.
Change-Id: I621a9ec198c31e28a383818efeb4b3f835ba1d6f
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Other plugins which want to use JGit nullness annotations in their code
cannot do this if the annotations aren't part of the published API.
Unfortunately it looks like although Eclipse JDT allows to use custom
nullness annotation types per project, it does not understand if those
annotations are used mixed with other nullness annotations in other
projects. E.g. EGit can either configure JGit annotations for NPE
analysis and so "understand" nullness from JGit API but so it loses the
ability to use any other nullness annotations to annotate its own code.
Change-Id: Ieeeb578c2fe35223a7561d668dce8e767dc89ef0
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Update the project-specific Eclipse settings to replace the use of the
org.eclipse.jdt.annotation.Nullable class the new JGit-specific
@Nullable annotation. I verified that Eclipse reports errors when the
return value of a method annotated with
@org.eclipse.jgit.annotations.Nullable is dereferenced without a null
check.
Also remove the Maven and MANIFEST.MF dependencies on
org.eclipse.jdt.annotation.
Eclipse null analysis uses three annotations: @Nullable, @NonNull and
@NonNullByDefault. All three are updated in this patch because it is
invalid to set the Eclipse preferences to empty values. So far only
@Nullable has been introduced in org.eclipse.jgit.annotations.
My personal preference is to follow the advice in Effective Java and
avoid the null-return idiom, and to avoid passing null values in
general. This sets the expectation is that arguments and return types
are assumed non-null unless otherwise documented. If that is the
expectation, then consistent application of @NonNull is redundant and
hurts readability by cluttering the code, obscuring the occasional
@Nullable annotation that really requires attention.
If the JGit community decides there is value in using the @NonNull and
@NonNullByDefault annotations we can add them--this change configures
Eclipse to use them.
Change-Id: I9af1b786d1b44b9b0d9c609480dc842df79bf698
Signed-off-by: Terry Parker <tparker@google.com>
This header was removed unintentionally from some bundles in
3a4a5a4e57. Restore it to ensure lazy
activation of bundles.
Change-Id: I1f841f978fb93278e3ec0533a01f1363510dd976
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In Bug 476164 it was reported that EGit doesn't start when the platform
comes with jsch 0.1.51 while this version of EGit/JGit brings jsch
0.1.53. This could be caused by outdated uses-clauses. Hence recompute
them using PDE tooling.
Bug: 476164
Change-Id: I185ba097884ead9cd034eba842bd3bf34181a99b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If UploadPack or ReceivePack has an exception record an identifier
associated with the repository as part of the log message. This can
help the HTTP admin track down the offending repository and take
action to repair the root cause.
Change-Id: I58f22b33cdb40994f044a26fba9fe965b45be51d
Hooks are now obtained via a convenient API like git commands, and
callers don't have to check for their existence.
The pre-commit hook has been updated accordingly.
Change-Id: I3383ffb10e2f3b588d7367b9139b606ec7f62758
Signed-off-by: Laurent Delaigue <laurent.delaigue@obeo.fr>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-3.7:
Add log4j and slf4j-log4j bridge to jgit feature
Use slf4j to log instead of printing to System.err
Use Target Platform Definition DSL to generate target platforms
Change-Id: Ic8779868150c910fa55fd20348e35723e6add0f1
Core classes to parse and process .gitattributes files including
support for reading attributes in WorkingTreeIterator and the
dirCacheIterator.
The implementation follows the git ignore implementation. It supports
lazy reading attributes while walking the working tree.
Bug: 342372
CQ: 9078
Change-Id: I05f3ce1861fbf9896b1bcb7816ba78af35f3ad3d
Also-by: Marc Strapetz <marc.strapetz@syntevo.com>
Also-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Also-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Signed-off-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
The current IgnoreRule/FileNameMatcher implementation scales not well
with huge repositories - it is both slow and memory expensive while
parsing glob expressions (bug 440732). Addtitionally, the "double star"
pattern (/**/) is not understood by the old parser (bug 416348).
The proposed implementation is a complete clean room rewrite of the
gitignore parser, aiming to add missing double star pattern support and
improve the performance and memory consumption.
The glob expressions from .gitignore rules are converted to Java regular
expressions (java.util.regex.Pattern). java.util.regex.Pattern code can
evaluate expression from gitignore rules considerable faster (and with
less memory consumption) as the old FileNameMatcher implementation.
CQ: 8828
Bug: 416348
Bug: 440732
Change-Id: Ibefb930381f2f16eddb9947e592752f8ae2b76e1
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
It's an internal package which isn't part of the API. Mark it x-internal
to silence @since tag warnings which are only raised for new API.
Bug: 440757
Change-Id: Id05deaca43f135cd1bfe83cf1f29787cbbdbecac
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This move avoids that all consumers of org.eclipse.jgit depend on Apache
httpclient. Also add another feature to make this optional for OSGi
consumers as well.
Change-Id: I5ef5e00c53678b9e1d7cfd54bbca3ff6f1c1c967
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This change implements the http connection abstraction with the help of
org.apache.http.client.HttpClient. The default implementation used by
JGit is still the JDK HttpURLConnection. But now JGit users have the
possibility to switch completely to org.apache.httpclient. The reason
for this is that in certain (e.g. cloud) environments you are forced to
use the org.apache classes.
Change-Id: I0b357f23243ed13a014c79ba179fa327dfe318b2
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Previously all HTTP communication was done with the help of
java.net.HttpUrlConnection. In order to make JGit usable in environments
where the direct usage of such connections is not allowed but where the
environment provides other means to get network connections an
abstraction for connections is introduced. The idea is that new
implementations of this interface will be introduced which will not use
java.net.HttpUrlConnection but use e.g.
org.apache.client.http.HttpClient to provide network connections.
One example: certain cloud infrastructures don't allow that components
in the cloud communicate directly with HttpUrlConnection. Instead they
provide services where a component can ask for a connection (given a
symbolic name for the destination) and where the infrastructure returns
a preconfigured org.apache.http.client.HttpClient. In order to allow
JGit to be running in such environments we need the abstraction
introduced in this commit.
Change-Id: I3b06629f90a118bd284e55bb3f6465fe7d10463d
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Package was renamed, so I had to update the imports. Also, I verified
bitmap serialization was still compatible.
Change-Id: I161ad3875b963b56001beab477ef8d072accee4f
This silences some discouraged access warnings issued since
TestRepository uses PackWriter which is in an internal package.
Change-Id: Ic9c4631e237c2fe1996c518328ecc2a9ab5c348b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This should prevent class cast problems caused by jgit and egit bundles
wiring to different versions of com.jcraft.jsch.
Bug: 420903
Change-Id: Icabe40209ea07369e2b7eee31952d131aef3fbf1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
DirCacheCheckout had a bug when the parentdirectory of a worktree was a
symlink. DirCacheCheckout was deleting those symlinks under certain
conditions. This was fixed in I81735ba0394ef6794e9b2b8bdd8bd7e8b9c6460f
without a test because previously it was hard to setup tests containing
symlinks.
BUG: 412489
Change-Id: I2513166af519d6fc01d1eae3976ad6cff6f98530
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This breaks all existing callers once. Applications are not supposed
to build against the internal storage API unless they can accept API
churn and make necessary updates as versions change.
Change-Id: I2ab1327c202ef2003565e1b0770a583970e432e9
A pack bitmap index is an additional index of compressed
bitmaps of the object graph. Furthermore, a logical API of the index
functionality is included, as it is expected to be used by the
PackWriter.
Compressed bitmaps are created using the javaewah library, which is a
word-aligned compressed variant of the Java bitset class based on
run-length encoding. The library only works with positive integer
values. Thus, the maximum number of ObjectIds in a pack file that
this index can currently support is limited to Integer.MAX_VALUE.
Every ObjectId is given an integer mapping. The integer is the
position of the ObjectId in the complete ObjectId list, sorted
by offset, for the pack file. That integer is what the bitmaps
use to reference the ObjectId. Currently, the new index format can
only be used with pack files that contain a complete closure of the
object graph e.g. the result of a garbage collection.
The index file includes four bitmaps for the Git object types i.e.
commits, trees, blobs, and tags. In addition, a collection of
bitmaps keyed by an ObjectId is also included. The bitmap for each entry
in the collection represents the full closure of ObjectIds reachable
from the keyed ObjectId (including the keyed ObjectId itself). The
bitmaps are further compressed by XORing the current bitmaps against
prior bitmaps in the index, and selecting the smallest representation.
The XOR'd bitmap and offset from the current entry to the position
of the bitmap to XOR against is the actual representation of the entry
in the index file. Each entry contains one byte, which is currently
used to note whether the bitmap should be blindly reused.
Change-Id: Id328724bf6b4c8366a088233098c18643edcf40f
Otherwise loading javax.net.ssl.TrustManager fails if
osgi.compatibility.bootdelegation=false which became the Equinox default
since bug 344850 was fixed.
Bug: 392056
Change-Id: I464871723649095942dbf77da93890ac8ec39075
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Chris Aniszczyk <zx@twitter.com>
This prevents a lot of unnecessary warnings about disouraged usage of
the org.eclipse.jgit.internal package within JGit itself.
Change-Id: Ia6683902809425fd7245e7d5d344c2ff8f317ebb
The package was removed in I763590a45d75f00a09097ab6f89581a3bbd3c797
Change-Id: Ifa9e75714f85d17609f9bf61581aaed0631a6fa7
Signed-off-by: Kevin Sawicki <kevin@github.com>
Adds the following commands:
- Add
- Init
- Status
- Sync
- Update
This also updates AddCommand so that file patterns added that
are submodules can be staged in the index.
Change-Id: Ie5112aa26430e5a2a3acd65a7b0e1d76067dc545
Signed-off-by: Kevin Sawicki <kevin@github.com>
Signed-off-by: Chris Aniszczyk <zx@twitter.com>
In practice the DHT storage layer has not been performing as well as
large scale server environments want to see from a Git server.
The performance of the DHT schema degrades rapidly as small changes
are pushed into the repository due to the chunk size being less than
1/3 of the pushed pack size. Small chunks cause poor prefetch
performance during reading, and require significantly longer prefetch
lists inside of the chunk meta field to work around the small size.
The DHT code is very complex (>17,000 lines of code) and is very
sensitive to the underlying database round-trip time, as well as the
way objects were written into the pack stream that was chunked and
stored on the database. A poor pack layout (from any version of C Git
prior to Junio reworking it) can cause the DHT code to be unable to
enumerate the objects of the linux-2.6 repository in a completable
time scale.
Performing a clone from a DHT stored repository of 2 million objects
takes 2 million row lookups in the DHT to locate the OBJECT_INDEX row
for each object being cloned. This is very difficult for some DHTs to
scale, even at 5000 rows/second the lookup stage alone takes 6 minutes
(on local filesystem, this is almost too fast to bother measuring).
Some servers like Apache Cassandra just fall over and cannot complete
the 2 million lookups in rapid fire.
On a ~400 MiB repository, the DHT schema has an extra 25 MiB of
redundant data that gets downloaded to the JGit process, and that is
before you consider the cost of the OBJECT_INDEX table also being
fully loaded, which is at least 223 MiB of data for the linux kernel
repository. In the DHT schema answering a `git clone` of the ~400 MiB
linux kernel needs to load 248 MiB of "index" data from the DHT, in
addition to the ~400 MiB of pack data that gets sent to the client.
This is 193 MiB more data to be accessed than the native filesystem
format, but it needs to come over a much smaller pipe (local Ethernet
typically) than the local SATA disk drive.
I also never got around to writing the "repack" support for the DHT
schema, as it turns out to be fairly complex to safely repack data in
the repository while also trying to minimize the amount of changes
made to the database, due to very common limitations on database
mutation rates..
This new DFS storage layer fixes a lot of those issues by taking the
simple approach for storing relatively standard Git pack and index
files on an abstract filesystem. Packs are accessed by an in-process
buffer cache, similar to the WindowCache used by the local filesystem
storage layer. Unlike the local file IO, there are some assumptions
that the storage system has relatively high latency and no concept of
"file handles". Instead it looks at the file more like HTTP byte range
requests, where a read channel is a simply a thunk to trigger a read
request over the network.
The DFS code in this change is still abstract, it does not store on
any particular filesystem, but is fairly well suited to the Amazon S3
or Apache Hadoop HDFS. Storing packs directly on HDFS rather than
HBase removes a layer of abstraction, as most HBase row reads turn
into an HDFS read.
Most of the DFS code in this change was blatently copied from the
local filesystem code. Most parts should be refactored to be shared
between the two storage systems, but right now I am hesistent to do
this due to how well tuned the local filesystem code currently is.
Change-Id: Iec524abdf172e9ec5485d6c88ca6512cd8a6eafb
Matthias Sohn