JGit's Nullable type was added[1] in the hope of being able to add
nullness annotations that (a) do not preclude building and running
with Java 7 and (b) could be shared by Gerrit, which uses a custom
Nullable type for other reasons[2]. Sharing a type is useful because
Eclipse's null analysis is only able to use one Nullable type at a
time in a given workspace (so for this analysis to function in a
workspace used to develop Gerrit, JGit and Gerrit would need to use
the same Nullable type).
The new Nullable type has CLASS instead of RUNTIME retention because
there wasn't any obvious use for the annotation at run time.
Gerrit uses the Nullable annotation to communicate with Guice. Guice
injection happens at runtime, so it needs to be able to read the
@Nullable annotations at run time[3]. Otherwise Guice produces
provisioning errors, such as
3) null returned by binding at com.google.gerrit.lucene.LuceneChangeIndex$Factory.create()
but parameter 7 of com.google.gerrit.lucene.LuceneChangeIndex.<init>() is not @Nullable
Switch to RUNTIME retention to avoid this.
While at it, update the javadoc to explain more clearly how this
annotation relates to other Nullable types[4]. This should make it
clearer why JGit needed another Nullable type:
A. Avoiding dependency on Java 8
B. RUNTIME retention to allow Guice to read the annotation at run time
C. Named Nullable so Guice can recognize the annotation
D. Not an addition to Java EE's javax.annotation package, to avoid
the split-package problem[2] that prevents the annotation from
being readable at run time when loaded from an OSGi container
E. Avoiding heavyweight dependencies, deprecated dependencies, and
dependencies on package internals
org.checkerframework.checker.nullness.qual.Nullable: A
com.sun.istack.internal.Nullable: B, E
*.CheckForNull, *.NullAllowed, etc: C
edu.umd.cs.findbugs.annotations.Nullable: B, E
javax.annotation.Nullable: D
org.eclipse.jdt.annotation.Nullable: B
org.jetbrains.annotations.Nullable: B
org.jmlspecs.annotation.Nullable: E
android.annotation.Nullable, android.support.annotation.Nullable: E
[1] https://git.eclipse.org/r/59993
[2] https://gerrit-review.googlesource.com/50112
[3] https://github.com/google/guice/blob/master/core/src/com/google/inject/internal/Nullability.java
[4] https://github.com/typetools/checker-framework/blob/5832a01f1/checker/src/org/checkerframework/checker/nullness/NullnessAnnotatedTypeFactory.java#L118http://types.cs.washington.edu/checker-framework/current/checker-framework-manual.html#nullness-related-work
Change-Id: I6c482653d2b53e3509abb11211b67fc29cf2949c
Signed-off-by: Jonathan Nieder <jrn@google.com>
If the password char array is null constructing a new String from this
array fails with a NPE. Add a null check to fix this.
Change-Id: Ifae6eecca38d5f114861f44658a32521e6e96866
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This should mirror the behavior of `git push --atomic` where the
client asks the server to apply all-or-nothing. Some JGit servers
already support this based on a custom DFS backend. InMemoryRepository
is extended to support atomic push for unit testing purposes.
Local disk server side support inside of JGit is a more complex animal
due to the excessive amount of file locking required to protect every
reference as a loose reference.
Change-Id: I15083fbe48447678e034afeffb4639572a32f50c
PreReceiveHook is given a Collection<ReceiveCommand> and it can be
very useful here to call ReceiveCommand.filter(cmds, NOT_ATTEMPTED).
Overload filter to accept both Iterable and List.
Keep backwards binary compatibility for List by upcasting to Iterable.
Change-Id: Ib1341876c703670945ef209edc8259715ee86c26
Recursively copying a tree into a DirCache is a bottleneck for some
algorithms like the in memory merge code in Gerrit Code Review. Drop
a layer down in the stack and use CanonicalTreeParser directly as the
addition logic only processes 1 tree at a time and does not need the
merge sorting feature (or overhead) of TreeWalk.
Combined with 761814fe9c ("DirCacheEntry: Speed up creation by
avoiding string cast") tree loading 38,900 entries nearly halves
in running time from 70ms to 36ms on some platforms.
Change-Id: If1490ca25de0679a71cf508f59b486f9cc816165
Application code sometimes wants to read a DirCache from an ObjectId,
but its confusing how to do this because its buried inside the
DirCacheBuilder.
Use this utility in a few places within JGit that also want to read
a DirCache from a tree's ObjectId.
Change-Id: I578b7e18e58753d154937f4ab835012b09e5adca
Instead of checking every entry for .gitattributes only look for the
entry on request by TreeWalk. This avoids impacting uses like RevWalk
filtering history.
When the attrs is requested skip to the start of the tree and look for
.gitattributes until either it is found, or it is impossible to be
present. Due to the sorting rules of tree entries .gitattributes
should be among the first or second entries in the tree so very few
entries will need to be considered.
Waiting to find the .gitattributes file by native ordering may miss
attrs for files like .config, which sorts before .gitattributes.
Starting from the front of the tree on demand ensures the attributes
are parsed as early as necessary to process any entry in the tree.
Due to TreeWalk recursively processing up the tree of iterators we
cannot just reset the current CanonicalTreeParser to the start as
parent parsers share the same path buffer as their children.
Resetting a parent to look for .gitattributes may overwrite path
buffer data used by a child iterator. Work around this by building a
new temporary CanonicalTreeParser instance.
Change-Id: Ife950253b687be325340d27e9915c9a40df2641c
Line breaking before , is ugly to read. Most formatters and humans
expect line break after , so update a few offensive locations.
Use String.format() for the construction of the error message when
a bad DirCachEntry is being failed on. This simplifies the code and
its not a performance critical section.
Change-Id: I5d990389e7ba24ef0861cf8ec0026ed030d4aeda
The checkPath function is available as a byte[] form, in fact the
String form just converts to byte[] to run the algorithm.
Having DirCacheEntry take a byte[] -> String -> byte[] to check if
each path is valid is a huge waste of CPU time. On some systems it
can double the time required to read 38,999 files from trees to the
DirCache. This slows down any operation using a DirCache.
Expose the byte[] form and use it for DirCacheEntry creation.
Change-Id: I6db7bc793ece99ff3c356338d793c07c061aeac7
The internal array may be longer than entryCnt, in this case the tail
of the array is padded with null entries. Do not return those to the
caller of getEntriesWithin().
Change-Id: I19efb05e103fab6b739ced407f6e28155a48dba6
If defined in .gitattributes call smudge filter during checkout.
To support checkout where current HEAD,index do not contain attributes
we need to also consider attributes from the tree we checkout. Therefore
CanonicalTreeParser has to learn how to provide attributes.
Change-Id: I168fdb81a8e1a9f991587b3e95a36550ea845f0a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When filters are defined for certain paths in gitattributes make
sure that clean filters are processed when adding new content to the
object database.
Change-Id: Iffd72914cec5b434ba4d0de232e285b7492db868
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Attributes represents a semantic collector of Attribute(s) and replaces
the anonymous Map<String,Attribute>. This class will be returned by
TreeWalk.getAttributes(). It offers convenient access to the attributes
wrapped in the Attributes object. Adds preparations for a future
Attribute Macro Expansion
Change-Id: I8348c8c457a2a7f1f0c48050e10399b0fa1cdbe1
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
When FS.runProcess was called and an InputStream was given the method
tried to pump the whole InputStream to the process. When the method
ended the InputStream was not giving any data anymore. Consequently
close the InputStream inside the method.
Change-Id: I0ed738a775e5c977b21447d195acee1ecf5e2cb9
When running from a non-bare repository, "jgit repo" checks whether
the rev passed in is a sha1 or branch name and in the latter case will
check out origin/<branch>.
We are expecting refs/remotes/origin/<branch>, but as a side effect of
using getRef we also end up looking for refs/origin/<branch>,
refs/heads/origin/<branch>, and so on. Avoid that by using exactRef
instead.
Signed-off-by: Jonathan Nieder <jrn@google.com>
Change-Id: I670b2e48a88138a1f2104ea201baa958e9edbddb
The Repository class provides only one method to look up a ref by
name, getRef. If I request refs/heads/master and that ref does not
exist, getRef will look further in the search path:
ref/refs/heads/master
refs/heads/refs/heads/master
refs/remotes/refs/heads/master
This behavior is counterintuitive, needlessly inexpensive, and usually
not what the caller expects.
Allow callers to specify whether to use the search path by providing
two separate methods:
- exactRef, which looks up a ref when its exact name is known
- findRef, which looks for a ref along the search path
For backward compatibility, keep getRef as a deprecated synonym for
findRef.
This change introduces findRef and exactRef but does not update
callers outside tests to use them yet.
Change-Id: I35375d942baeb3ded15520388f8ebb9c0cc86f8c
Signed-off-by: Jonathan Nieder <jrn@google.com>
org.eclipse.jgit.lib.Repository class is an example of the API which
should be written with Java 8 java.util.Optional<T> type. Unfortunately
this API is already released and widely used. The good clients are
currently doing their best with checking return values for null and bad
clients do not know how bad their code is.
I've tried not to change any logic and to be as less intrusive as
possible. Most of the JGit code was well prepared to this, only few
classes needed some smaller fixes.
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Because annotating getDirectory() and getFS() as Nullable would cause
lot of additional changes in JGit and EGit they are postponed.
Change-Id: Ie8369d2c9c5fac5ce83b3b1b9bc217d7b55502a3
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
File, FileInputStream and friends may throw FileNotFoundException even
if the file is existing e.g. when file permissions don't allow to access
the file content. In most cases this is a severe error we should not
suppress hence rethrow the FileNotFoundException in this case.
This may also fix bug 451508.
Bug: 451508
Change-Id: If4a94217fb5b7cfd4c04d881902f3e86193c7008
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The class FS_Win32 was always trying out to create a temporary symlink
in order to find out whether symlinks are supported. FS_Win32_Cygwin was
overwriting this method and always returned true. But when the user
running JGit does not have administrative rights then the creation of
symlinks is forbidden even if he is running on FS_Win32_Cygwin. A lot of
tests failed only on the Windows platform because of this. It was
correctly detected that FS_Win32_Cygwin is the filesystem abstraction to
be used but creation of symlinks always failed because of lacking
privileges of the user running the tests.
This fix teaches FS_Win32_Cygwin to behave like FS_Win32 and to test
whether symlinks can be created in order to find out whether symlinks
are supported.
Change-Id: Ie2394631ffc4c489bd37c3ec142ed44bbfcac726
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This fixes a warning saying this filter isn't needed anymore.
Change-Id: If77056378befe86c1773950dbe48a82c833fd532
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
getRef() loops over its search path to find a ref:
Ref ref = null;
for (String prefix : SEARCH_PATH) {
ref = readRef(prefix + needle, packed);
if (ref != null) {
ref = resolve(ref, 0, null, null, packed);
break;
}
}
fireRefsChanged();
return ref;
If readRef returns null (indicating that the ref does not exist), the
loop continues so we can find the ref later in the search path. And
resolve should never return null, so if we return null it should mean
we exhausted the entire search path and didn't find the ref.
... except that resolve can return null: it does so when it has
followed too many symrefs and concluded that there is a symref loop:
if (MAX_SYMBOLIC_REF_DEPTH <= depth)
return null; // claim it doesn't exist
Continue the loop instead of returning null immediately. This makes
the behavior more consistent.
Arguably getRef should throw an exception when a symref loop is
detected. That would be a more invasive change, so if it's a good
idea it will have to wait for another patch.
Change-Id: Icb1c7fafd4f1e34c9b43538e27ab5bbc17ad9eef
Signed-off-by: Jonathan Nieder <jrn@google.com>
If the index file exists but can't be read for example because of wrong
filesystem permissions we should throw a specific exception. This allows
EGit to handle this error situation.
Bug: 482607
Change-Id: I50bfcb719c45caac3cb5550a8b16307c2ea9def4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
According to [1] the pre-push hook expects two parameters which provide
the name and location of the destination remote, if a named remote is
not being used both values should be the same.
We did set the first parameter to null in that case which caused
ProcessBuilder to throw a NullPointerException since its start() method
doesn't accept null arguments.
[1] https://git-scm.com/docs/githooks#_pre_push
Bug: 482393
Change-Id: Idb9b0a48cefac01abfcfdf00f6d173f8fa1d9a7b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When checking out a branch we need to access the attributes stored
in the tree to be checked out. E.g. directly after a clone we checkout
the remote HEAD. In this case index and workingtree are still empty.
So we have to search the tree to be checked out for attributes.
Change-Id: I6d96f5d095ed2e3c259d4b12124e404f5215bd9f
Adds the getAttributes feature to the tree walk. The computation of
attributes needs to be done by the TreeWalk since it needs both a
WorkingTreeIterator and a DirCacheIterator.
Bug: 342372
CQ: 9120
Change-Id: I5e33257fd8c9895869a128bad3fd1e720409d361
Signed-off-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Other plugins which want to use JGit nullness annotations in their code
cannot do this if the annotations aren't part of the published API.
Unfortunately it looks like although Eclipse JDT allows to use custom
nullness annotation types per project, it does not understand if those
annotations are used mixed with other nullness annotations in other
projects. E.g. EGit can either configure JGit annotations for NPE
analysis and so "understand" nullness from JGit API but so it loses the
ability to use any other nullness annotations to annotate its own code.
Change-Id: Ieeeb578c2fe35223a7561d668dce8e767dc89ef0
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
The annotation is required for example in Repository case (patch
follows), where almost all non-void return methods return Nullable
except few returning NonNull. I definitely do not favor this style, but
it is a nightmare to clients to guess if the null check is needed or
not.
Change-Id: Ib2a778a246c6d84b7c32565f54df2385b59f6498
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
When JGit tries to detect symlink support the attempt to create a
symlink may fail with a java.lang.InternalError. This was reported for a
case where the application using JGit runs in Windows XP compatibility
mode using Oracle JDK 1.8. Handle this by assuming symlinks are not
supported in this case.
Bug: 471027
Change-Id: I978288754dea0c6fffd3457fad7d4d971e27c6c2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.1:
Prepare 4.1.2-SNAPSHOT builds
JGit v4.1.1.201511131810-r
Fallback exactRef: Do not ignore symrefs to unborn branch
RefDirectory.exactRef: Do not ignore symrefs to unborn branch
Change-Id: I66afb303f355aad8a7eaa7a6dff06de70ae9c490
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When asked to read a symref pointing to a branch-yet-to-be-born (such
as HEAD in a newly initialized repository), getRef and getRefs provide
different results.
getRef: SymbolicRef[HEAD -> refs/heads/master=00000000]
getRefs and getAdditionalRefs: nothing
exactRef should match the getRef behavior: it is meant to be a
simpler, faster version of getRef that lets you search for a ref
without resolving it using the search path without other semantic
changes. But the fallback implementation of exactRef relies on getRefs
and produces null for this case.
Luckily the in-tree RefDatabase implementations override exactRef and
get the correct behavior. But any out-of-tree storage backend that
doesn't inherit from DfsRefDatabase or RefDirectory would still return
null when it shouldn't.
Let the fallback implementation use getRef instead to avoid this.
This means that exactRef would waste some effort traversing the ref
search path when the named ref is not found --- but subclasses tend to
override exactRef for performance already, so in the default
implementation correctness is more important.
Bug: 478865
Change-Id: I60f04e3ce3bf4731640ffd2433d329e621330029
When attempting to determine the size of a blob that does not exist,
the RenameDetector throws a MissingObjectException.
The fix is to return a size of zero if the size is requested for a blob
id that doesn't exist.
Bug: 481577
Change-Id: I4e86276039c630617610cc51d0eefa56d7d3952f
Signed-off-by: Rüdiger Herrmann <ruediger.herrmann@gmx.de>
When asked to read a symref pointing to a branch-yet-to-be-born (such
as HEAD in a newly initialized repository), DfsRepository and
FileRepository return different results.
FileRepository:
exactRef("HEAD") => null
DfsRepository:
exactRef("HEAD") => SymbolicRef[HEAD -> refs/heads/master=00000000]
getRef("HEAD") returns the same as DfsRepository's exactRef in both
backends.
The intended behavior is the DfsRepository one: exactRef() is supposed
to be like getRef(), but more exact because it doesn't need to
traverse the search path.
The discrepancy is because DfsRefDatabase implements exactRef()
directly with the intended semantics, while RefDirectory uses a
fallback implementation built on top of getRefs(). getRefs() skips
symrefs to an unborn branch.
Override the fallback implementation with a correct implementation
that is similar to getRef() to avoid this. A followup change will fix
the fallback.
Change-Id: Ic138a5564a099ebf32248d86b93e2de9ab3c94ee
Reported-by: David Pursehouse <david.pursehouse@sonymobile.com>
Improved-by: Christian Halstrick <christian.halstrick@sap.com>
Bug: 478865
This reverts commit bbcbcab8d3.
These classes were part of the public API and should not be removed
until JGit 5.0.
Change-Id: Ife4bee69f82151de6ef8ea1a4c6c146d91bbf0d5
These classes make improper use of internal classes in the public API
and were replaced by corresponding classes in the JGit 4.1 release.
Change-Id: I3d474210e49089aa788314b4e08f505f0d26619b
Signed-off-by: Terry Parker <tparker@google.com>
These API changes were introduced in f523f21e and 73474466.
BitmapBuilder is an interface to be implemented by implementors of this
interface. According to OSGi semantic versioning rules breaking API
changes require update of the minor version only if implementors of the
API have to be adapted and the changes do not affect clients of the API.
Change-Id: If45d204181ea9bc788b6b57693ca17b1847564c7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
PackWriterBitmapPreparer (which is in another package) is already well
aware of the mapping between EWAHCompressedBitmaps and the
higher-level CompressedBitmap objects of the BitmapIndexImpl API.
Making the CompressedBitmap type public makes the translation more
obvious and wouldn't break any abstractions that aren't already
broken. So expose it.
This is all under org.eclipse.jgit.internal so there are no API
stability guarantees --- we can change the API if internals change
(for example if some day there are bitmaps spanning multiple packs).
In particular this means the confusing toBitmap helper can be removed.
Change-Id: Ifb2e8804a6d5ee46e82a76d276c4f8507eaa2a4c
Update the project-specific Eclipse settings to replace the use of the
org.eclipse.jdt.annotation.Nullable class the new JGit-specific
@Nullable annotation. I verified that Eclipse reports errors when the
return value of a method annotated with
@org.eclipse.jgit.annotations.Nullable is dereferenced without a null
check.
Also remove the Maven and MANIFEST.MF dependencies on
org.eclipse.jdt.annotation.
Eclipse null analysis uses three annotations: @Nullable, @NonNull and
@NonNullByDefault. All three are updated in this patch because it is
invalid to set the Eclipse preferences to empty values. So far only
@Nullable has been introduced in org.eclipse.jgit.annotations.
My personal preference is to follow the advice in Effective Java and
avoid the null-return idiom, and to avoid passing null values in
general. This sets the expectation is that arguments and return types
are assumed non-null unless otherwise documented. If that is the
expectation, then consistent application of @NonNull is redundant and
hurts readability by cluttering the code, obscuring the occasional
@Nullable annotation that really requires attention.
If the JGit community decides there is value in using the @NonNull and
@NonNullByDefault annotations we can add them--this change configures
Eclipse to use them.
Change-Id: I9af1b786d1b44b9b0d9c609480dc842df79bf698
Signed-off-by: Terry Parker <tparker@google.com>
Update existing @Nullable uses to use the JGit-internal version of the
annotation.
Change-Id: I95234ffad4c3110f9597fbd3a2760f653e22ecf7
Signed-off-by: Terry Parker <tparker@google.com>
Commit 847b3d1 enabled annotation-based NPE analysis to JGit.
In so doing, it introduced a new dependency on the org.eclipse.jdt that
is undesirable. Follow Gerrit's lead by adding an internal Nullable type
(see
https://gerrit.googlesource.com/gerrit/+/stable-2.11/gerrit-common/src/main/java/com/google/gerrit/common/Nullable.java).
The javax.annotations.Nullable class uses a retention policy of RUNTIME,
whereas the org.eclipse.jdt.annotation.Nullable class used a policy of
CLASS. Since I'm not aware of tools that make use of the annotation at
runtime I chose the CLASS policy. If in the future there is a benefit to
retaining the annotations in the running binary we can update this
class.
Change-Id: I63dc8f9a6dc46b517cbde211bb5e2f8521a54d04
Signed-off-by: Terry Parker <tparker@google.com>
A CompressedBitmap represents a pair (EWAH bit vector, PackIndex
assigning bit positions to git objects). The bit vector is a member
field and the PackIndex is implicit via the 'this' reference to the
outer class.
Make this clearer by making CompressedBitmap a static class and
replacing the 'this' reference by an explicit field.
Likewise for CompressedBitmapBuilder.
Change-Id: Id85659fc4fc3ad82034db3370cce4cdbe0c5492c
Suggested-by: Terry Parker <tparker@google.com>
When creating bitmaps during gc, the bitmaps in tipCommitBitmaps are
built in setupTipCommitBitmaps using the following procedure:
0. create a bitmap ('reuse') that lists all ancestors of commits
whose existing bitmaps will be reused. I will call this the
reused part of history.
1. initialize a bitmap for each of the pack's "want"s by taking
a copy of the 'reuse' bitmap and setting the bit corresponding
to the one wanted commit.
2. walk through ancestors of wants, excluding the reused part of
history. Add parents of visited commits to bitmaps that have
those commits.
3. AND-NOT each tipCommitBitmap against the 'reuse' bitmap
4. Sort the bitmaps and AND-NOT each against the previous so they
partition the new commits.
The OR against 'reuse' in step 1 and the AND-NOT against 'reuse'
cancel each other out, except when commits from the reused part of
history are added to a bitmap in step 2. So avoid adding commits from
the reused part of history in step 2 and skip the OR and AND-NOT.
Performance impact (thanks to Terry for measuring):
The initial "selecting bitmaps" phase of garbage collection decreased
from (83 + 81 + 85) / 3 = 83 to (56 + 57 + 56) / 3 = 56.3, meaning
nearly a ~50% speedup of that phase.
Tested-by: Terry Parker <tparker@google.com>
Change-Id: I26ea695809594347575d14a1d8e6721b8608eb9c
Instead of using the newRevFilter helper, call the appropriate
RevFilter constructor directly. This means one less hop to find
documentation about what the RevFilter will do.
Change-Id: Ida6ff1c0457a47a1bd1e4ed0fd1dd42a616d214f
When setupTipCommitBitmaps is called, writeBitmaps does not have any
bitmaps saved, so these calls to .add always add a single commit and
do not OR in a bitmap.
The objects returned by nextObject after a commit walk is finished
are trees and blobs. Non-commit objects do not have bitmaps
associated so the call to .add also can only add a single object.
Change-Id: I7b816145251a7fa4f1fffe0d03644257ea7f5440
* changes:
Remove BitmapRevFilter.getCountOfLoadedCommits
Make BitmapBuilder.getBitmapIndex public
Deprecate BitmapBuilder.add and introduce simpler addObject method
Add @Override annotations to BitmapIndexImpl
Rely on bitmap RevFilter to filter walk during bitmap selection
Use 'reused' bitmap to filter walk during bitmap selection
Rely on bitmap RevFilter to filter tip commit setup walk
Use 'reused' bitmap to filter tip commit setup walk
Include ancestors of reused bitmap commits in reuse bitmap again
This is the caller that the BitmapBuilder.add method was designed
around. Moving away from .add makes it more verbose but hopefully
clearer.
Change-Id: I57b1d7c1dc8fb800b242b76c606922b5aa36b9b2
This puts the code for include() in each RevFilter returned by
newRevFilter in one place and should make the code easier to
understand and modify.
Change-Id: I590ac4604b61fc1ffe7aba2ed89f8139847ccac3
The count of loaded commits is equal to the number of commits returned
by the walk. Simplify BitmapRevFilter by counting them in the caller.
Change-Id: Ia95da47831d9e89d6f8068470ec4529aaabfa7dd
Every Bitmap in current JGit code has an associated BitmapIndex. Make
it public in BitmapBuilder to make retrieving bitmaps to OR in from
that index easier.
Change-Id: I2773aa94d8b67f12194608e6317c0792a5de21e2
The BitmapIndex.BitmapBuilder.add API is subtle:
/**
* Adds the id and the existing bitmap for the id, if one
* exists, to the bitmap.
*
* @return true if the value was not contained or able to be
* loaded.
*/
boolean add(AnyObjectId objectId, int type);
Reading the name of the method does not make it obvious what it will
do. Does it add the named object to the bitmap, or all objects
reachable from it? It depends on whether the BitmapIndex owns an
existing bitmap for that object. I did not notice this subtlety when
skimming the javadoc, either. This resulted in enough confusion to
subtly break the bitmap building code (see change
I30844134bfde0cbabdfaab884c84b9809dd8bdb8 for details).
So discourage use of the add() API by deprecating it.
To replace it, provide a addObject() method that adds a single object.
This way, callers can decide whether to use addObject() or or() based
on the context.
For example,
if (bitmap.add(c, OBJ_COMMIT)) {
for (RevCommit p : c.getParents()) {
rememberToAlsoHandle(p);
}
}
can be replaced with
if (bitmap.contains(c)) {
// already included
} else if (index.getBitmap(c) != null) {
bitmap.or(index.getBitmap(c));
} else {
bitmap.addObject(c, OBJ_COMMIT);
for (RevCommit p : c.getParents()) {
rememberToAlsoHandle(p);
}
}
which is more verbose but makes it clearer that the behavior
depends on the content of index.getBitmaps().
Change-Id: Ib745645f187e1b1483f8587e99501dfcb7b867a5
This makes it easier to distinguish between implementations of methods
from the interface from helpers internal to org.eclipse.jgit.internal.storage.*.
This was illegal in Java 5 but JGit requires newer Java these days.
Change-Id: I92c65f3407a334acddd32ec9e09ab7d1d39c4dc6
This RevWalk filters out reused bitmap commits via the 'reuse' bitmap.
Avoid possible wasted time and complexity by not also redundantly
marking them UNINTERESTING.
Change-Id: Ibb714002ddac599963d148a9aab90645fcc73141
When building fullBitmap in order to determine which ancestor chain to
add this commit to, we were excluding the ancestors of reusedCommits
using markUninteresting. This use of markUninteresting is a bit
wasteful because we already have a bitmap indicating exactly which
commits should be excluded (which can save some walking). Use it.
A separate commit will remove the now-redundant markUninteresting
call.
No behavior change intended (except for performance improvement).
Change-Id: I1112641852d72aa05c9a8bd08a552c70342ccedb
This RevWalk filters out reused bitmap commits via the 'reuse' bitmap.
Avoid possible wasted time and complexity by not redundantly marking
them UNINTERESTING any more.
Change-Id: If467ccd1d75e17cf9367b2a0399fca3f9d52adf9
When garbage collecting, we decide to reuse some bitmaps in older
history from the previous pack to save time. The remainder of commit
selection only involves commits not covered by those bitmaps.
Currently we carry that out in two ways:
1. by building a bitmap representing the already-covered commits,
for easy containment checks and AND-NOT-ing against
2. by marking the reused bitmap commits as uninteresting in the
RevWalk that finds new commits
The mechanism in (2) is less efficient than (1): rw.next() will walk
back from reused bitmap commits to check whether the commit it is
about to emit is an ancestor of them, when using the bitmap from (1)
would let us perform the same check with a single contains() call.
Add a RevFilter teaching the RevWalk to perform that same check
directly using the bitmap from (1).
The next time the RevWalk is used, a different RevFilter is installed
so this does not break that.
A later commit will drop the markUninteresting calls.
No functional change intended except a possible speedup.
Change-Id: Ic375210fa69330948be488bf9dbbe4cb0d069ae6
Prior to this change, DfsInserter would not insert an object into a pack
if it already existed in another pack in the repository, even if that
pack was unreachable. Consider this sequence of events:
- Object FOO is pushed to a repository.
- Subsequent ref changes make FOO UNREACHABLE_GARBAGE.
- FOO is subsequently re-inserted using a DfsInserter, but skipped
due to existing in UNREACHABLE_GARBAGE.
- The repository is repacked; FOO will not be written into a new pack
because it is not yet reachable from a reference. If the
UNREACHABLE_GARBAGE packs are deleted, FOO disappears.
- A reference is updated to reference FOO. This reference is now broken
as FOO was removed when the repacking process deleted the
UNREACHABLE_GARBAGE pack that stored the only copy of FOO.
The garbage collector can't safely delete the UNREACHABLE_GARBAGE
pack because FOO might be in the middle of being re-inserted/re-packed.
This change writes a duplicate copy of an object if it only exists in
UNREACHABLE_GARBAGE. This "freshens" the object to give it a chance to
survive long enough to be made reachable through a reference.
Change-Id: I20f2062230f3af3bccd6f21d3b7342f1152a5532
Signed-off-by: Mike Williams <miwilliams@google.com>
Until 320a4142 (Update bitmap selection throttling to fully span
active branches, 2015-10-20), setupTipCommitBitmaps contained code
along the following lines:
for (PackBitmapIndexRemapper.Entry entry : bitmapRemapper) {
if (!reuse(entry))
continue;
RevCommit rc = (RevCommit) rw.peel(rw.parseAny(entry));
reuseCommits.add(new BitmapCommit(rc));
EWAHCompressedBitmap bitmap =
remap.ofObjectType(remap.getBitmap(rc), OBJ_COMMIT);
writeBitmaps.addBitmap(rc, bitmap, 0);
reuse.add(rc, OBJ_COMMIT);
}
writeBitmaps.clearBitmaps(); // Remove temporary bitmaps
This loop OR-ed together bitmaps for commits whose bitmaps would be
reused. A subtle point is the use of the add() method, which ORs in a
bitmap from the BitmapIndex when it exists and falls back to OR-ing in
a single bit when that bitmap does not exist in the BitmapIndex.
Commit 320a4142 removed the addBitmap step, so the bitmap does not
exist in the BitmapIndex and the fallback behavior is triggered.
Simplify and restore the intended behavior by avoiding use of the
subtle use of the add() method --- use or() directly instead.
Change-Id: I30844134bfde0cbabdfaab884c84b9809dd8bdb8
When packing is able to reuse lots of deltas from existing packs, those
objects are marked as "doNotAttemptDelta" and do not contribute to
DeltaTask's computeTopPaths() "totalWeight" calculation.
In the extreme case when all packs are reusable, "totalWeight" will be
zero. DeltaTask.partitionTasks() uses "totalWeight" to determine a
"weightPerThread" size it uses to set up DeltaTasks. When "totalWeight"
is small, partitionTasks() ends up creating a DeltaTask for every
unique path.
For a large repository, the small "weightPerThread" can result in the
creation of >100k tasks (for the MSM 3.10 Linux repository, the count
was ~150k). This makes the "task stealing" mechanism in DeltaTask very
inefficient, because every attempt to steal work does a linear walk
through all tasks, searching for the one with the most work remaining,
which is O(N^2) comparisons. For the MSM 3.10 repository when all
deltas were reusable, PackWriter.parallelDeltaSearch() took
(1615+1633+1458)/3 = 1568 seconds.
The error is that DeltaTask treats the weights of objects marked as
"doNotAttemptDelta" inconsistently. It ignores the weights when
calculating "totalWeight" but uses them when partitioning the tasks.
The fix is to also ignore them when partitioning the tasks.
With this patch applied, PackWriter.parallelDeltaSearch() on the
MSM 3.10 repository when all deltas are reused went from taking
1568 seconds to 62ms (>25k speedup).
This patch also fixes a totalWeight initialization error in
DeltaTask.computeTopPaths().
Change-Id: I2ae37efa83bca42b0e716266ae6aa9d182e76d9c
Signed-off-by: Terry Parker <tparker@google.com>
Avoid leaving the reader in suspense by handling the unusual
(!RevCommit) case first. As a nice side effect, there is less nesting
to keep track of in the rest of the loop body.
No functional change intended.
Change-Id: I1580de444fccde08070f696218c12041151a924a
When the file <git-dir>/hooks/pre-push exists make sure that is is
executing during a push. The pre-push hook runs during git push, after
the remote refs have been updated but before any objects have been
transferred.
Change-Id: Ibbb58ee3227742d1a2f913134ce11e7a135c7f4c
In order to support filters in gitattributes FS.runProcess() is made
public. Support for stdin redirection has been added. Support for binary
data on stdin/stdout (as used be clean/smudge filters) has been added.
Change-Id: Ice2c152e9391368dc5748d7b825a838e3eb755f9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fixed random errors in discoverGitSystemConfig() on Linux where the
process error stream was closed by readPipe() before or while
GobblerThread was reading from it.
Marked readPipe() as @Nullable and fixed potential NPE in
discoverGitSystemConfig() on readPipe() return value.
Fixed process error output randomly mixed with other threads log
messages.
Change-Id: Id882af2762cfb75f010f693b2e1c46eb6968ee82
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Add comments and rename variables in PackWriterBitmapPreparer to improve
readability.
Change-Id: I49e7a1c35307298f7bc32ebfc46ab08e94290125
Signed-off-by: Terry Parker <tparker@google.com>
Java compiler must generate synthetic access methods for private methods
and fields of the enclosing class if they are accessed from inner
classes and vice versa.
While invisible in the code, those synthetic access methods exist in the
bytecode and seem to produce some extra execution overhead at runtime
(compared with the direct access to this fields or methods), see
https://git.eclipse.org/r/58948/.
By removing the "private" access modifier from affected methods and
fields we help compiler to avoid generation of synthetic access methods
and hope to improve execution performance.
To validate changes, one can either use javap or use Bytecode Outline
plugin in Eclipse. In both cases one should look for "synthetic
access$<number>" methods at the end of the class and inner class files
in question - there should be none.
NB: don't mix this "synthetic access$" methods up with "public synthetic
bridge" methods generated to allow generic method override return types.
Change-Id: I94fb481b68c84841c1db1a5ebe678b13e13c962b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Java compiler must generate synthetic access methods for private methods
and fields of the enclosing class if they are accessed from inner
classes and vice versa.
While invisible in the code, those synthetic access methods exist in the
bytecode and seem to produce some extra execution overhead at runtime
(compared with the direct access to this fields or methods), see
https://git.eclipse.org/r/58948/.
By removing the "private" access modifier from affected methods and
fields we help compiler to avoid generation of synthetic access methods
and hope to improve execution performance.
To validate changes, one can either use javap or use Bytecode Outline
plugin in Eclipse. In both cases one should look for "synthetic
access$<number>" methods at the end of the class and inner class files
in question - there should be none.
NB: don't mix this "synthetic access$" methods up with "public synthetic
bridge" methods generated to allow generic method override return types.
Change-Id: Ie7b65f251ec4452d5a5ed48aa0f272cf49a9aecd
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Java compiler must generate synthetic access methods for private methods
and fields of the enclosing class if they are accessed from inner
classes and vice versa.
While invisible in the code, those synthetic access methods exist in the
bytecode and seem to produce some extra execution overhead at runtime
(compared with the direct access to this fields or methods), see
https://git.eclipse.org/r/58948/.
By removing the "private" access modifier from affected methods and
fields we help compiler to avoid generation of synthetic access methods
and hope to improve execution performance.
To validate changes, one can either use javap or use Bytecode Outline
plugin in Eclipse. In both cases one should look for "synthetic
access$<number>" methods at the end of the class and inner class files
in question - there should be none.
NB: don't mix this "synthetic access$" methods up with "public synthetic
bridge" methods generated to allow generic method override return types.
Change-Id: I0ebaeb2bc454cd8051b901addb102c1a6688688b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Java compiler must generate synthetic access methods for private methods
and fields of the enclosing class if they are accessed from inner
classes and vice versa.
While invisible in the code, those synthetic access methods exist in the
bytecode and seem to produce some extra execution overhead at runtime
(compared with the direct access to this fields or methods), see
https://git.eclipse.org/r/58948/.
By removing the "private" access modifier from affected methods and
fields we help compiler to avoid generation of synthetic access methods
and hope to improve execution performance.
To validate changes, one can either use javap or use Bytecode Outline
plugin in Eclipse. In both cases one should look for "synthetic
access$<number>" methods at the end of the class and inner class files
in question - there should be none.
NB: don't mix this "synthetic access$" methods up with "public synthetic
bridge" methods generated to allow generic method override return types.
Change-Id: If53ec94145bae47b74e2561305afe6098012715c
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Expose the following bitmap selection parameters via PackConfig:
"bitmapContiguousCommitCount", "bitmapRecentCommitCount",
"bitmapRecentCommitSpan", "bitmapDistantCommitSpan",
"bitmapExcessiveBranchCount", and "bitmapInactiveBranchAge".
The value of bitmapContiguousCommitCount, whereby bitmaps are
created for the most recent N commits in a branch, has never
been verified. If experiments show that they are not valuable,
then we can simplify the implementation so that there is only
a concept of recent and distant commit history (defined by
"bitmapRecentCommitCount"), and the only controls we need are
"bitmapRecentCommitSpan" and "bitmapDistantCommitSpan".
Change-Id: I288bf3f97d6fbfdfcd5dde2699eff433a7307fb9
Signed-off-by: Terry Parker <tparker@google.com>
Replace the “bitmapCommitRange” parameter that was recently introduced
with two new parameters: “bitmapExcessiveBranchCount” and
“bitmapInactiveBranchAgeInDays”. If the count of branches does not
exceed “bitmapExcessiveBranchCount”, then the current algorithm is kept
for all branches.
If the branch count is excessive, then the commit time for the tip
commit for each branch is used to determine if a branch is “inactive”.
"Active" branches get full commit selection using the existing
algorithm. "Inactive" branches get fewer bitmaps near the branch tips.
Introduce a "contiguousCommitCount" parameter that always enforces that
the N most recent commits in a branch are selected for bitmaps. The
previous nextSelectionDistance() algorithm created anywhere from 1-100
contiguous bitmaps at branch tips.
For example, consider a branch with commits numbering 0-300, with 0
being the most recent commit. If the most recent 200 commits are not
merge commits and the 200th commit was the last one selected,
nextSelectionDistance() returned 100, causing commits 200-101 to be
ignored. Then a window of size 100 was evaluated, searching for merge
commits. Since no merge commits are found, the next commit (commit 0)
was selected, for a total of 1 commit in the topmost 100 commits.
If instead the 250th commit was selected, then by the same logic
commit 50 is selected. At that point nextSelectionDistance() switches to
selecting consecutive commits, so commits 0-50 in the topmost 100
commits are selected. The "contiguousCommitCount" parameter provides
more determinism by always selecting a constant number or topmost
commits.
Add an optimization to break out of the inner loop of selectCommits() if
all of the commits for the current branch have already been found.
When reusing bitmaps from an existing pack, remove unnecessary
populating and clearing of the writeBitmaps/PackBitmapIndexBuilder.
Add comments to PackWriterBitmapPreparer, rename methods and variables
for readability.
Add tests for bitmap selection with and without merge commits and with
excessive branch pruning triggered.
Note: I will follow up with an additional change that exposes the new
parameters through PackConfig.
Change-Id: I5ccbb96c8849f331c302d9f7840e05f9650c4608
Signed-off-by: Terry Parker <tparker@google.com>
Since Git 2.6 wildcard restrictions for refspecs have been loosened:
refspecs like "refs/heads/*foo:refs/heads/foo*" are valid now.
See Git commit 8d3981ccbed9fc211b4e67105015179d9d2a5692
Change-Id: Icb78afbd282c425173b3a7bc10eadc4015689bb8
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Building on top of https://git.eclipse.org/r/#/c/56391/
Here we preserve compatibility with JetS3t
and add 2 new native JGit encryption implementations.
For reference, see connection configuration files:
* Version 0: jgit-s3-connection-v-0.properties
* Version 1: jgit-s3-connection-v-1.properties
* Version 2: jgit-s3-connection-v-2.properties
Change-Id: I713290bcacbe92d88e5ef28ce137de73dd1abe2f
Signed-off-by: Andrei Pozolotin <andrei.pozolotin@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
See previous attempt: https://git.eclipse.org/r/#/c/16674/
Here we preserve as much of JetS3t mode as possible
while allowing to use new Java 8+ PBE algorithms
such as PBEWithHmacSHA512AndAES_256
Summary of changes:
* change pom.xml to control long tests
* add WalkEncryptionTest.launch to run long tests
* add AmazonS3.Keys to to normalize use of constants
* change WalkEncryption to support AES in JetS3t mode
* add WalkEncryptionTest to test remote encryption pipeline
* add support for CI configuration for live Amazon S3 testing
* add log4j based logging for tests in both Eclipse and Maven build
To test locally, check out the review branch, then:
* create amazon test configuration file
* located your home dir: ${user.home}
* named jgit-s3-config.properties
* file format follows AmazonS3 connection settings file:
accesskey = your-amazon-access-key
secretkey = your-amazon-secret-key
test.bucket = your-bucket-for-testing
* finally:
* run in Eclipse: WalkEncryptionTest.launch
* or
* run in Shell: mvn test --define test=WalkEncryptionTest
Change-Id: I6f455fd9fb4eac261ca73d0bec6a4e7dae9f2e91
Signed-off-by: Andrei Pozolotin <andrei.pozolotin@gmail.com>
If the checkout path is currently a non-empty directory (and was a link
or a regular file before), this directory will be removed before
performing checkout, but only if the checkout path is specified.
Bug: 474973
Change-Id: Ifc6c61592d9b54d26c66367163acdebea369145c
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
A bitmap index contains bitmaps for a set of commits in a pack file.
Creating a bitmap for every commit is too expensive, so heuristics
select the most "important" commits. The most recent commits are the
most valuable. To clone a repository only those for the branch tips are
needed. When fetching, only commits since the last fetch are needed.
The commit selection heuristics generally work, but for some
repositories the number of selected commits is prohibitively high. One
example is the MSM 3.10 Linux kernel. With over 1 million commits on
2820 branches, the current heuristics resulted in +36k selected commits.
Each uncompressed bitmap for that repository is ~413k, making it
difficult to complete a GC operation in available memory.
The benefit of creating bitmaps over the entire history of a repository
like the MSM 3.10 Linux kernel isn't clear. For that repository, most
history for the last year appears to be in the last 100k commits.
Limiting bitmap commit selection to just those commits reduces the count
of selected commits from ~36k to ~10.5k. Dropping bitmaps for older
commits does not affect object counting times for clones or for fetches
on clients that are reasonably up-to-date.
This patch defines a new "bitmapCommitRange" PackConfig parameter to
limit the commit selection process when building bitmaps. The range
starts with the most recent commit and walks backwards. A range of 10k
considers only the 10000 most recent commits. A range of zero creates
bitmaps only for branch tips. A range of -1 (the default) does not limit
the range--all commits in the pack are used in the commit selection
process.
Change-Id: Ied92c70cfa0778facc670e0f14a0980bed5e3bfb
Signed-off-by: Terry Parker <tparker@google.com>
On a server also running Gerrit that is using RepoCommand to
convert from an XML manifest to a git submodule superproject
periodically, it would be handy to be able to use Gerrit's
submodule subscription feature[1] to update the superproject
automatically between RepoCommand runs as changes are merged
in each subprojects.
This requires setting the 'branch' field for each submodule
so that Gerrit knows what branch to watch. Add an option to
do that.
Setting the branch field also is useful for plain Git users,
since it allows them to use "git submodule update --remote" to
manually update all submodules between RepoCommand runs.
[1] https://gerrit-review.googlesource.com/Documentation/user-submodules.html
Change-Id: I1a10861bcd0df3b3673fc2d481c8129b2bdac5f9
Signed-off-by: Stefan Beller <sbeller@google.com>
* stable-4.1:
pgm: Open RevWalk and TreeWalk in try-with-resource
ant: Open Repository and Git in try-with-resource
pgm: Create instances of Git in try-with-resource
FanoutBucket: Create ObjectInserter.Formatter in try-with-resource
Fix compiler warnings in DiffFormatter.writeGitLinkText
Change-Id: I448ecc9a1334977d9f304dd61ea20c7a8e692b10
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Clirr doesn't support Java 8 hence use japicmp instead.
See https://github.com/siom79/japicmp
Change-Id: If4b30a6d6aa849b4d6b3b0c900558c609822840c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This patch makes JGit parsing of ignore rules containing unmatched '['
bracket compatible to the Git CLI.
Since '[' starts character group, Git tries to parse the ignore rule as
a shell glob pattern and if the character group is not closed, the glob
pattern is invalid and so the ignore rule never matches anything.
See also http://article.gmane.org/gmane.comp.version-control.git/278699.
Bug: 478490
Change-Id: I734a4d14fcdd721070e3f75d57e33c2c0700d503
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
- Remove declaration of IOException that is no longer thrown
- Add missing //$NON-NLS-1$ to prevent "Non-externalized string literal"
warning.
These warnings seem to have been introduced by If13f7b406.
Change-Id: I30058eed31b92067a6ab22e787732b08e29f8d63
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
Since 4.0 we require Java 7 so there is no longer a need to override the
following methods in FS_POSIX, FS_Win32, FS_Win32_Cygwin
- lastModified()
- setLastModified()
- length()
- isSymlink()
- exists()
- isDirectory()
- isFile()
- isHidden()
Hence implement these methods in FS and remove overrides in subclasses.
Change-Id: I5dbde6ec806c66c86ac542978918361461021294
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
As discussed on https://git.eclipse.org/r/53836 it does not make sense
to have two similar utility classes in same package with intersecting
functionality. To not break the API, all methods from FileUtil are
copied to FileUtils, all FileUtil API is made deprecated and redirecting
now to FileUtils. Moved simple methods which are available in Java 7 API
are made package private and can be removed at any point later entirely
(right now they are in use).
Bug: 475070
Change-Id: Idffcf9840496c448173af7c052d8898ada68e27b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
SystemReader.isMacOs() and SystemReader.isWindows() return values are
unlikely to change during the JVM lifetime (except tests). Don't read
system properties each time the methods are called, just use previously
calculated value.
Change-Id: I495521f67a8b544e7b7247d99bbd05a42ea16d20
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
An attempt to re-implement not well documented Git CLI behavior for
patterns with backslashes.
It looks like Git silently ignores all \ characters in ignore rules, if
they are NOT covered by 3 cases described in [1]:
{quote}
1) ... Put a backslash ("\") in front of the first hash for patterns
that begin with a hash.
...
2) Trailing spaces are ignored unless they are quoted with backslash
("\").
...
3) Put a backslash ("\") in front of the first "!" for patterns that
begin with a literal "!", for example, "\!important!.txt".
{quote}
Undocumented also is the fact that backslash itself can be escaped by
backslash.
So \h\e\l\l\o\.t\x\t rule matches hello.txt and a\\\\b a\b in Git CLI.
Additionally, the glob parser [2] knows special meaning of backslash:
{quote}
One can remove the special meaning of '?', '*' and '[' by preceding
them by a backslash, or, in case this is part of a shell command
line, enclosing them in quotes. Between brackets these characters
stand for themselves. Thus, "[[?*\]" matches the four characters
'[', '?', '*' and '\'.
{quote}
[1] https://www.kernel.org/pub/software/scm/git/docs/gitignore.html
[2] http://man7.org/linux/man-pages/man7/glob.7.html
Bug: 478065
Change-Id: I3dc973475d1943c5622103701fa8cb3ea0684e3e
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Currently we fail to properly recognize character group if the pattern
before character group contains opening bracket.
See comment from Sebastien Arod on https://git.eclipse.org/r/56678/
Change-Id: I70d3657a2a328818ea2bdc1409d18ecb3a85825b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Current DiffFormat behavior regarding submodules (aka git links) is
incorrect. The "Subproject commit <sha1>" appears as part of the diff
header, rather than as its own hunk.
--> From JGit implementation
diff --git a/plugins/cookbook-plugin b/plugins/cookbook-plugin
index b9d3ca8..ec6ed89 160000
--- a/plugins/cookbook-plugin
+++ b/plugins/cookbook-plugin
-Subproject commit b9d3ca8a65030071e28be19296ba867ab424fbbf
+Subproject commit ec6ed89c47ba7223f82d9cb512926a6c5081343e
--> From C Git 2.5.2
diff --git a/plugins/cookbook-plugin b/plugins/cookbook-plugin
index b9d3ca8..ec6ed89 160000
--- a/plugins/cookbook-plugin
+++ b/plugins/cookbook-plugin
@@ -1 +1 @@
-Subproject commit b9d3ca8a65030071e28be19296ba867ab424fbbf
+Subproject commit ec6ed89c47ba7223f82d9cb512926a6c5081343e
The current way of processing submodules results in no hunk header and
includes the contents of the hunk as part of the headers. To fix this, we
can't just have our writeGitLinkDiffText output the hunk header. We have
to change the flow so that the raw text gets parsed as a diff. The easiest
way to do this is to fake the RawText in the FormatResult when we have a
GITLINK.
It should be noted that it seems possible for there to be a difference
between a GITLINK and a non-GITLINK, but I don't think this can happen in
practice, so I don't think we need to worry too much about it.
This patch also fixes up the test for GitLink headers, as the test was
for the old behavior. My setup has 3 other failing tests which may or
may not be the result of environmental changes. However, the same tests
fail without this commit, so I do not believe they are related.
Bug: 477759
Change-Id: If13f7b406904fad814416c93ed09ea47ef183337
Signed-off-by: Jacob Keller <jacob.keller@gmail.com>
Ignore rules should escape $^(){}+| chars if using regular expressions,
because they should be treated literally if they aren't part of a
character group.
Bug: 478055
Change-Id: Ic7276442d7f8f02594b85eae1ef697362e62d3bd
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
If a client mistakenly tries to send a tag object as a shallow line
JGit blindly assumes this is a commit and tries to parse the tag
buffer using the commit parser. This can cause an obtuse error like:
InvalidObjectIdException: Invalid id: t c0ff331234...
The "t" comes from the "object c0ff331234..." line of the tag tring
to be parsed as though it where the "tree" line of a commit.
Run any client supplied shallow lines through the RevWalk to lookup
the object types. Fail fast with a protocol exception if any of them
are non-commit.
Skip objects not known to this repository. This matches behavior
with git-core's upload-pack, which sliently skips over any shallow
line object named by the client but not known by the server.
Change-Id: Ic6c57a90a42813164ce65c2244705fc42e84d700
Properties.containsKey() is the correct call here; contains() was testing
if a value is present but the key is what was meant.
Change-Id: Ice72c9f4388583e18cf8aca6e837cc4299fd07fd
When we have a URI that contains an empty path component (that is
it only contains a "/") we want to fall back to the host as
humanish name.
This change is according to the behavior of upstream git, which
falls back on the hostname when guessing directory names for
newly cloned repositories (see [1] for the discussion).
[1] http://article.gmane.org/gmane.comp.version-control.git/274669
Change-Id: I44400c6ab72a2722d2155d53d63671bd867d6c44
Signed-off-by: Patrick Steinhardt <ps@pks.im>
If no refs match the input list and we are writing to a batch,
the returned new commit from write() will match the current commit.
Adding a command to the batch for this case is harmless as it will
succeed, but it's more straightforward to just skip adding a command
in this case.
Add tests or the combination of saving matching refs and saving to a
batch.
Change-Id: I6837389b08e6c80bc2d4c9e9c506d07293ea5fb2
This header was removed unintentionally from some bundles in
3a4a5a4e57. Restore it to ensure lazy
activation of bundles.
Change-Id: I1f841f978fb93278e3ec0533a01f1363510dd976
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
In Bug 476164 it was reported that EGit doesn't start when the platform
comes with jsch 0.1.51 while this version of EGit/JGit brings jsch
0.1.53. This could be caused by outdated uses-clauses. Hence recompute
them using PDE tooling.
Bug: 476164
Change-Id: I185ba097884ead9cd034eba842bd3bf34181a99b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
It was reported in [1] that 197e3393a5 led
to a performance regression in a BFG benchmark. Analysis showed that
this is caused by the exists() method in FS_POSIX, now overriding the
default implementation in FS. The default implementation of FS.exists()
uses java.io.File.exists(), while the new implementation in FS_POSIX
uses java.nio.file.Files.exists() - by simply removing the override in
FS_POSIX, performance was restored.
Profiling showed that java.nio.file.Files.exists() is substantially
slower than java.io.File.exists(), to the point where the exists() call
doubles the average cost of a call to
ObjectDirectory.insertUnpackedObject() - which the BFG uses a lot,
because it's rewriting history. Average times measured on Ubuntu were:
java.io.File.exists() - 4 microseconds
java.nio.file.Files.exists() - 60 microseconds
The loose object exists test should be using java.io.File and not FS.
ObjectDirectory uses FS.resolve() to traverse symlinks to objects but
then once inside objects all 256 sharded directories should be real
directories, and the object files should be real files, not dangling
symlinks. java.io.File.exists() is sufficient here, and faster.
Change ObjectDirectory to use File.exists() once its computed the File
handle.
This does mean JGit cannot run ObjectDirectory code on an abstract
virtual filesystem plugged into NIO2. If you really want to run JGit on
an esoteric non-standard filesystem like "in memory" you should look at
the DFS storage backend, which has fewer abstraction points to deal
with. Or write your own from scratch.
[1] https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg02954.html
Change-Id: I74684dc3957ae1ca52a7097f83a6c420aa24310f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
- add @since annotation for new API method
- silence non-externalized String warning
Change-Id: I864176ced64e9569e7f2cdf8f777720655bfc578
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
On a local filesystem the packed-refs file will be orphaned if it is
replaced by another client while the current client is reading the old
one. However, since NFS servers do not keep track of open files, instead
of orphaning the old packed-refs file, such a replacement will cause the
old file to be garbage collected instead. A stale file handle exception
will be raised on NFS servers if the file is garbage collected (deleted)
on the server while it is being read. Since we no longer have access to
the old file in these cases, the previous code would just fail. However,
in these cases, reopening the file and rereading it will succeed (since
it will reopen the new replacement file). So retrying the read is a
viable strategy to deal with stale file handles on the packed-refs file,
implement such a strategy.
Since it is possible that the packed-refs file could be replaced again
while rereading it (multiple consecutive updates can easily occur with
ref deletions), loop on stale file handle exceptions, up to 5 extra
times, trying to read the packed-refs file again, until we either read
the new file, or find that the file no longer exists. The limit of 5 is
arbitrary, and provides a safe upper bounds to prevent infinite loops
consuming resources in a potential unforeseen persistent error
condition.
Change-Id: I085c472bafa6e2f32f610a33ddc8368bb4ab1814
Signed-off-by: Martin Fick<mfick@codeaurora.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add a public API to the FileUtils to determine if an IOException is a
stale NFS file handle exception. This will make it easier to detect
such errors, and interpret them consistently throughout the codebase.
This new API is a bit more lenient in its detection than the previous
detection, and should be able to detect some errors which previously
were not identified as stale file handle exceptions because they had the
word NFS in the error message. Adjust the packfile handling code to use
this new API for detection.
Change-Id: I21f80014546ba1afec7335890e5ae79e7f521412
Signed-off-by: Martin Fick<mfick@codeaurora.org>
Signaling the need to flush() only via the interrupted status of a
copying thread doesn't work realiably with jsch. The write() method of
com.jcraft.jsch.Session catches the InterruptedException in several
places. As a result StreamCopyThread can easily miss the interrupt if it
was interrupted during the dst.write() or dst.flush() call. When it
happens, StreamCopyThread will not send some data to the remote side and
will not get the response back, because remote side will wait for more
data from us.
The flushCount field incremented during flush() method guarantees we
don't miss flush() even if jsch catches InterruptedException in
dst.write() or dst.flush() calls.
Checking the flushCount after dst.write() is needed because dst.write()
can clear interrupt status, in this case the next blocking src.read()
won't throw an exception and we will not call flush().
Flush is performed only after src.read() was blocked and thrown an
InterruptedIOException exception, this guarantees that we flush all the
data available in src so far (src.read() doesn't block while more is
available).
FlushCount is reset to 0 only when there were no flush() calls since
last blocked read, that means we flushed all data available in src. If
there were flush() calls, the interrupt status is restored, so next
blocked read will throw InterruptedException and we will flush()
again.
Change-Id: I692b226edaff502f06235ec05da9052b5fe6478a
Signed-off-by: Dmitry Neverov <dmitry.neverov@gmail.com>
There should be no functional change, the logic updated only to make
code simple so that compiler can understand what is going for. Removed
all @SuppressWarnings("null") annotations since they cannot be used if
"org.eclipse.jdt.core.compiler.problem.potentialNullReference" option is
set to the "error" level.
Bug: 470647
Change-Id: Ie93c249fa46e792198d362e531d5cbabaf41fdc4
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
- use NIO2's Files.move() to reimplement rename()
- provide a second method accepting CopyOptions which can be used to
request atomic move.
Change-Id: Ibcf722978e65745218a1ccda45344ca295911659
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Root commits are commits with zero parents. If a commmit has no
parents it is the first commit in the repository. In general the root
commits should be unique for any given project, as the first commit
will be created at a different time, by a different user with its own
message. These root commits can be used as a "fingerprint" to
identify disjoint histories.
Change-Id: Id891dbc1f17c816cea404569578bb7635ff85cdb
The FS class and the subclasses FS_POSIX assumed in the findHook()
method that every repository has a valid gitDir. But in tests when using
in-memory-repositories this is not true and this method was generating
NPEs. Change the method to return null if no repository directory can be
determined.
Change-Id: I38a4d36dc6452b5dacae3d0dbf562b569ca3c19b
DfsGarbageCollector asks PackWriter for the set of objects packed
after the bitmap index is written out. This is now null as it was
cleared to release memory. Instead use PackBitmapIndexBuilder to
build the set as it also has the objects.
Reduce memory in PackBitmapIndexBuilder by fully discarding the
ObjectToPack instances. This was the original intent of commit
4bb523475d ("PackWriter: shed memory while creating bitmaps")
but failed as the instances were still held live here.
Switch to BlockList instead of ObjectToPack[]. This allows the
JVM to allocate many smaller arrays instead of one contiguous
array with 5.2M reference pointers. In a tight heap the smaller
allocations are more feasible.
Reduce the initial EWAHCompressedBitmaps for the 4 type maps. On
average a typical repository is 30% commits, 30% trees and 30% blobs.
These bitmaps are typically very dense. PackWriter orders objects by
commit, tree, blob when writing the file so these should always be a
very dense run of 1s with some 0s before and after. So even the 1/3rd
allocation is likely too large, but the later trim() will reduce the
internal buffer anyway.
Change-Id: If0b80a31cb00894f1485ff8f53ef7ae5a759a046
Jonathan Nieder