The ReftableCompactor supported a byteLimit, but this is currently
unused. The FileReftableStack has a more sophisticated strategy that
amortizes compaction costs.
Rename min/maxUpdateIndex to reflogExpire{Min,Max}UpdateIndex to
reflect their purpose more accurately.
Since reflogs are generally pruned chronologically (oldest entries are
expired first), one can only prune entries on full compaction, so they
should not be set by default.
Rephrase the function Reader#minUpdateIndex and maxUpdateIndex. These
vars are documented to affect log entries, but semantically, they are
about ref entries. Since ref entries have their timestamps
delta-compressed, it is important for the min/maxUpdateIndex values to
be coherent between different tables.
The logical timestamps for log entries do not have to be coherent in
different tables, as the timestamps of a log entry is part of the key.
For example, a table written at update index 20 may contain a tombstone
log entry at timestamp 1.
Therefore, we set ReftableWriter's min/maxUpdateIndex from the merged
tables we are compacting, rather than from the compaction settings
(which should only control reflog expiry.)
The previous behavior could drop log entries erroneously, especially
in the presence of tombstone log entries. Unfortunately, testing this
properly requires both an API for adding log tombstones, and a more
refined API for controlling automatic compaction. Hence, no test.
Change-Id: I2f4eb7866f607fddd0629809e8e61f0b9097717f
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
The merged table contains handles to open files. A full compaction
causes those files to be closed, and so further lookups would fail
with EBADF.
Change-Id: I7bb74f7228ecc7fec9535b00e56a617a9c18e00e
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Commit 6216b0de changed the behavior of the setMirror(),
setCloneAllBranches(), and setBranchesToClone() operations. Before
that commit, these could be set and reset independently and only in
call() it would be determined what exactly to do. Since that commit,
the last of these calls would determine the operation. This means
that the sequence
cloneCommand.setCloneAllBranches(true);
cloneCommand.setBranchesToClone(/* some list of refs */);
would formerly do a "clone all" giving a fetch refspec with wildcards
+refs/heads/*:refs/remotes/origin/*
which picks up new upstream branches, whereas since commit 6216b0de
individual non-wildcard fetch refspecs would be generated and new
upstream branches would not be fetched anymore.
Undo this behavioral change. Make the operations independently settable
and resettable again, and determine the exact operation only in call():
mirror=true > cloneAll=true > specific refs, where ">" means "takes
precedence over", and if none is set assume cloneAll=true.
Note that mirror=true implies setBare(true).
Bug: 559796
Change-Id: I7162b60e99de5e3e512bf27ff4113f554c94f5a6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.5:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I73d16b53df02bf735c2431588143efe225a4b5b4
* stable-5.4:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I76a62da98182f0c504b1ea8b7d37cecdf4eea7e0
* stable-5.3:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I12002dbfed9dff14fc6d2df9787d92eab5b1fa78
* stable-5.2:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I790098be00ff3f9b8278d54ae4fb7c11311816e9
* stable-5.1:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Change-Id: I741059a1d0d5950ab5bc16ec70352655ee926a24
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The externalized error message added in f4fc640 ("BasePackConnection:
Check for expected length of ref advertisement", Dec 18, 2019) uses a
malformed string format. Since there is only one formatting argument,
it should be referenced with '{0}' rather than '{1}'.
Change-Id: Ibda864dfb0bb902fe07ae4bba73117b212046e8a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Since ObjectDatabase and PackFile don't know their repository use the
packfile's grand-grand-parent directory as an identifier for the
repository the packfile resides in.
Remove metric for a repository if the number of cached bytes for the
repository drops to 0 in order to ensure the map of cached bytes per
repository doesn't contain repositories which have no data cached in the
WindowCache.
Change-Id: I969ab8029db0a292e7585cbb36ca0baa797da20b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Java GC evicts all SoftReferences when the used heap size comes close to
the maximum heap size. This means peaks in heap memory consumption can
flush the complete WindowCache which was observed to have negative
impact on performance of upload-pack in Gerrit.
Hence add a boolean option core.packedGitUseStrongRefs to allow using
strong references to reference packfile pages cached in the WindowCache.
If this option is set to true Java gc can no longer flush the
WindowCache to free memory if the used heap comes close to the maximum
heap size. On the other hand this provides more predictable performance.
Bug: 553573
Change-Id: I9de406293087ab0fa61130c8e0829775762ece8d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.1:
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I17da3c5183eca536aa2be3972bc5df45c9d75f1b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Using exceptions during normal operations - for example with the
desire of expanding an array in the failure case - can have a
severe performance impact. When exceptions are instantiated,
a stack trace is collected. Generating stack trace can be expensive.
Compared to that, checking an array for length - even if done many
times - is cheap since this is a check that can run in just a
handful of CPU cycles.
Change-Id: Ifaf10623f6a876c9faecfa44654c9296315adfcb
Signed-off-by: Patrick Hiesel <hiesel@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.5:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I11f9a387ac3dc7d22a4f2e70bb8d89169b4e9afe
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.4:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I78902d5feecb2c09134b64ec2f3b48b2c3bab37d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.3:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I1b560b36d169cfa02cc5450ad0fa0bd85f9f42d8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.2:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I67a07d92718188bdf7f8a13b83e9f538ecf4b22f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Calling sh -c with a script path containing backslashes may fail since
the shell may try to process them as escape characters.
Instead of calling
sh.exe -c 'C:\path\script "$@"' 'C:\path\script' other args
call
sh.exe -c '$0 "$@"' 'C:\path\script' other args
which avoids this escape processing.
Note that this is not specific to Windows. If the path or the script
name contain backslashes, this also occurs on Unix.
Bug: 558577
Change-Id: I47d63db6f8644f956c55c42b07dbcad7d7f305aa
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
This will change the behavior in the CLI to resemble that of C-Git more
closely by printing the stderr of the hooks to the CLI stderr
independently of the exit code of the hook.
This is also useful for the corresponding EGIT-Change, which will add
the ability to show the hook output in eclipse.
With this also the stderr can be shown even if the exit code is 0.
Bug: 553471
Change-Id: Ie7bc503fe39e270e9b93dd1108b5879f02a12b4c
Signed-off-by: Tim Neumann <Tim.Neumann@advantest.com>
This is a preparation for the corresponding EGit change to remove
redundant output in the aborted by commit hook popup.
Bug: 553469
Change-Id: Id5f39a4df659fafc9d951668e6f53fed4380de9d
Signed-off-by: Tim Neumann <Tim.Neumann@advantest.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add the following statistics
- cache hit count and hit ratio
- cache miss count and miss ratio
- count of successful and failed loads
- rate of failed loads
- load, eviction and request count
- average and total load time
Use LongAdder instead of AtomicLong to implement counters in order to
improve scalability.
Optionally expose these metrics via JMX, they are registered with the
platform MBean server if the config option jmx.WindowCacheStats = true
in the user or system level git config.
Bug: 553573
Change-Id: Ia2d5246ef69b9c2bd594a23934424bc5800774aa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Git log supports the exclude pathspec, which allows for excluding paths
from the log command. JGit only supports adding paths to the log
command. See the following StackOverflow question for details:
https://stackoverflow.com/questions/59143934/java-jgit-how-to-get-git-
commits-not-affecting-certain-directories
This commit adds an excludePath() method to the log command. It does not
yet support regex or glob wildcards.
Change-Id: I8cd59950b87850b55a15c7e2ea5470145c9aae28
Signed-off-by: John Tipper <john_tipper@hotmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The reftable format supports fast inverse (SHA1 => ref) queries.
If the ref database does not support fast inverse queries, it may be
advantageous to build a complete SHA1 to ref map in advance for
multiple uses. To let applications decide, this function indicates
whether the inverse map is available.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Idaf7e01075906972ec21332cade285289619c2b3
This partially reverts Ic6bb5e66. It made private some methods in
ReceivePack that were protected before. At least two of those methods
(#init and #receivePackAndCheckConnectivity) are used by out-of-tree
subclasses for tracing and testing.
Make ReceivePack#init() and ReceivePack#receivePackAndCheckConnectivity
protected again to allow tracing and testing.
Change-Id: I7ff22c091fbfc2d94009c449d58e7c5ac9f4f759
Signed-off-by: Ivan Frade <ifrade@google.com>
When a conflicting file was blamed, JGit would not identify lines
coming from the merge parents. The main cause for this was that
Blame and BlameCommand simply added the first DirCacheEntry found
for a file to its queue of candidates (blobs or commits) to consider.
In case of a conflict this typically is the merge base commit, and
comparing a auto-merged contents against that base would yield
incorrect results.
Such cases have to be handled specially. The candidate to be
considered by the blame must use the working tree contents, but
at the same time behave like a merge commit/candidate with HEAD
and the MERGE_HEADs as parents. Canonical git does something very
similar, see [1].
Implement that and add tests.
I first did this for the JGit pgm Blame command. When I then tried
to do the same in BlameCommand, I noticed that the latter also
included some fancy but incomplete CR-LF handling. In order to
be able to use the new BlameGenerator.prepareHead() also in
BlameCommand this CR-LF handling was also moved into BlameGenerator
and corrected in doing so.
(Just considering the git config settings was not good enough,
CR-LF behavior can also be influenced by .gitattributes, and even
by whether the file in the index has CR-LF. To correctly determine
CR-LF handling for check-in one needs to do a TreeWalk with at
least a FileTreeIterator and a DirCacheIterator.)
[1] https://github.com/git/git/blob/v2.22.0/blame.c#L174
Bug: 434330
Change-Id: I9d763dd6ba478b0b6ebf9456049d6301f478ef7c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
David Pursehouse