The merged table contains handles to open files. A full compaction
causes those files to be closed, and so further lookups would fail
with EBADF.
Change-Id: I7bb74f7228ecc7fec9535b00e56a617a9c18e00e
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Using the batch cleanup operation via Source -> Cleanup -> "Use lambdas
where possible" from standard JDT.
Change-Id: I5452bd94fdccc920ade071228aeed3a8b9fdbe62
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
Commit 6216b0de changed the behavior of the setMirror(),
setCloneAllBranches(), and setBranchesToClone() operations. Before
that commit, these could be set and reset independently and only in
call() it would be determined what exactly to do. Since that commit,
the last of these calls would determine the operation. This means
that the sequence
cloneCommand.setCloneAllBranches(true);
cloneCommand.setBranchesToClone(/* some list of refs */);
would formerly do a "clone all" giving a fetch refspec with wildcards
+refs/heads/*:refs/remotes/origin/*
which picks up new upstream branches, whereas since commit 6216b0de
individual non-wildcard fetch refspecs would be generated and new
upstream branches would not be fetched anymore.
Undo this behavioral change. Make the operations independently settable
and resettable again, and determine the exact operation only in call():
mirror=true > cloneAll=true > specific refs, where ">" means "takes
precedence over", and if none is set assume cloneAll=true.
Note that mirror=true implies setBare(true).
Bug: 559796
Change-Id: I7162b60e99de5e3e512bf27ff4113f554c94f5a6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
When fetching remote objects, WalkFetchConnection searches remote
packs in the order provided by WalkRemoteObjectDatabase:getPackNames.
Previously, for TransportAmazonS3, the packs were in no particular
order. This resulted in potential many extra calls to get pack idx
files.
This change modifies TransportAmazonS3 and AmazonS3 so that
getPackNames returns a list sorted with the most recently modified
packs first. In the case of fetching recent changes to a repo,
this dramatically reduces the number of packs searched and speeds
up fetch.
Note: WalkRemoteObjectDatabase::getPackNames does not specify
the order of the returned names.
Testing: did "mvn clean install" in root dir and all tests passed.
And manually constructed some S3 repos and using jgit.sh
confirmed that the freshest pack was checked first.
Change-Id: I3b968fee825e793be55566e28c2d69d0cbe53807
Signed-off-by: Joshua Redstone <redstone@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.7:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I0538bcba259f7229790a602ac9de120464a1260d
* stable-5.6:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I79d615dff66493b60d3a4bcbdc57b9455e8d6673
* stable-5.5:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I73d16b53df02bf735c2431588143efe225a4b5b4
* stable-5.4:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I76a62da98182f0c504b1ea8b7d37cecdf4eea7e0
* stable-5.3:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I12002dbfed9dff14fc6d2df9787d92eab5b1fa78
* stable-5.2:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I790098be00ff3f9b8278d54ae4fb7c11311816e9
* stable-5.1:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Change-Id: I741059a1d0d5950ab5bc16ec70352655ee926a24
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The externalized error message added in f4fc640 ("BasePackConnection:
Check for expected length of ref advertisement", Dec 18, 2019) uses a
malformed string format. Since there is only one formatting argument,
it should be referenced with '{0}' rather than '{1}'.
Change-Id: Ibda864dfb0bb902fe07ae4bba73117b212046e8a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Since ObjectDatabase and PackFile don't know their repository use the
packfile's grand-grand-parent directory as an identifier for the
repository the packfile resides in.
Remove metric for a repository if the number of cached bytes for the
repository drops to 0 in order to ensure the map of cached bytes per
repository doesn't contain repositories which have no data cached in the
WindowCache.
Change-Id: I969ab8029db0a292e7585cbb36ca0baa797da20b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The default threads from the ForkJoinPool run without any privileges
when a SecurityManager is installed, leading to SecurityExceptions
when trying to create the probe file even if the application otherwise
has write privileges in the directory.
Use a dedicated ThreadPoolExecutor using daemon threads instead.
Bug: 551690
Change-Id: Id5376f09f0d7da5ceea367e1f0dfc70f823d62d3
Signed-off-by: Alex Jitianu <alex_jitianu@sync.ro>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Enable UnusedException at ERROR level which causes the build to fail
in many places with:
[UnusedException] This catch block catches an symbol and re-throws
another, but swallows the caught symbol rather than setting it as a
cause. This can make debugging harder.
Fix it by setting the caught exception as cause on the subsequently
thrown exception.
Note: The grammatically incorrect error message is copy-pasted as-is
from the version of ErrorProne currently used in Bazel; it has been
fixed by [1] in the latest version.
[1] https://github.com/google/error-prone/commit/d57a39c
Change-Id: I11ed38243091fc12f64f1b2db404ba3f1d2e98b5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If setInstance(SshSessionFactory) is called with parameter null
set default session factory using the newly introduced service loader
instead of hard-coding the default factory class.
Change-Id: I86b5932333cc53b706534a2822e0fd96e12e6e47
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use ServiceLoader and define
org.eclipse.jgit.transport.DefaultSshSessionFactory in
META-INF/services/org.eclipse.jgit.transport.SshSessionFactory so that
the legacy behavior is still the same.
Bug: 553625
Change-Id: Id1a65506140d921ed76d83699e3817f0d2ca08ed
Signed-off-by: Emmanuel Hugonnet <ehugonne@redhat.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Java GC evicts all SoftReferences when the used heap size comes close to
the maximum heap size. This means peaks in heap memory consumption can
flush the complete WindowCache which was observed to have negative
impact on performance of upload-pack in Gerrit.
Hence add a boolean option core.packedGitUseStrongRefs to allow using
strong references to reference packfile pages cached in the WindowCache.
If this option is set to true Java gc can no longer flush the
WindowCache to free memory if the used heap comes close to the maximum
heap size. On the other hand this provides more predictable performance.
Bug: 553573
Change-Id: I9de406293087ab0fa61130c8e0829775762ece8d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* added new public method getContent()
* changed checkoutEntry() to use the new getContent() method
* changed runExternalFilterCommand() parameter "entry" to "path" to be
independent from DirCacheEntry type
* removed runExternalFilterCommand() parameter "fs" as it can be derived
from repo
Bug: 356832
Change-Id: If8991b74870e0d8728093e98a990f93f1871b76a
Signed-off-by: Andre Bossert <andre.bossert@siemens.com>
* stable-5.1:
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I17da3c5183eca536aa2be3972bc5df45c9d75f1b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Using exceptions during normal operations - for example with the
desire of expanding an array in the failure case - can have a
severe performance impact. When exceptions are instantiated,
a stack trace is collected. Generating stack trace can be expensive.
Compared to that, checking an array for length - even if done many
times - is cheap since this is a check that can run in just a
handful of CPU cycles.
Change-Id: Ifaf10623f6a876c9faecfa44654c9296315adfcb
Signed-off-by: Patrick Hiesel <hiesel@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Protected field ReceivePack.connectivityChecker is of type
ConnectivityChecker which was located in internal package
org.eclipse.jgit.transport and was thus leaking an internal type.
In order to fix this move ConnectivityChecker to package
org.eclipse.jgit.transport.
Change-Id: Idc0af75502faf39f13a56157bbb6964d24720eb8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.6:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I4a77c602bfffed60535d0b8fc251a552b128068d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.5:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I11f9a387ac3dc7d22a4f2e70bb8d89169b4e9afe
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.4:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I78902d5feecb2c09134b64ec2f3b48b2c3bab37d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.3:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I1b560b36d169cfa02cc5450ad0fa0bd85f9f42d8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.2:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I67a07d92718188bdf7f8a13b83e9f538ecf4b22f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
SSLContext.getInstance("TLS") by default behaves differently on IBM
JDK than on Oracle or OpenJDK.[1] On IBM JDK one gets sockets that
have only TLSv1 enabled, which makes HTTPS connections fail since most
servers refuse this old protocol version. On Oracle JDK/OpenJDK, one
gets sockets with all available protocol versions enabled.
Explicitly enable all available TLS protocol versions to make
HTTPS connections work also on IBM JDK.
[1] https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/matchsslcontext_tls.html#matchsslcontext_tls
Bug: 558709
Change-Id: I5ffc57a78e67a6239b9dad54840a49a8ed28930a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.6:
Hooks: avoid problems with backslashes in paths
Change-Id: Ia9e4aca11de5a465b60898a37441a17133f6355b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Calling sh -c with a script path containing backslashes may fail since
the shell may try to process them as escape characters.
Instead of calling
sh.exe -c 'C:\path\script "$@"' 'C:\path\script' other args
call
sh.exe -c '$0 "$@"' 'C:\path\script' other args
which avoids this escape processing.
Note that this is not specific to Windows. If the path or the script
name contain backslashes, this also occurs on Unix.
Bug: 558577
Change-Id: I47d63db6f8644f956c55c42b07dbcad7d7f305aa
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
StringBuffer is synchronized which is slower and should be replaced with
StringBuilder according to its Javadoc.
Change-Id: If4d4a5a49da289ded34bbec97132ab7636b937cc
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
When converting to reftable format the option extensions.refStorage must
be set to "reftable" [1]. When converting back to refdir format this
config option needs to be removed.
Introduce constants for refStorage config options, also for the
"reftree" format.
[1] https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/master/Documentation/technical/reftable.md#Version-1
Change-Id: I190222fa5edc1ad7309daa9be17ca934ff7971e3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
and switch over strings where possible. Sometimes if statements are
chained and form a series of comparisons against constants. Using switch
statements improves readability.
Bug: 545856
Change-Id: Iacb78956ee5c20db4d793e6b668508ec67466606
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.6:
Add ability to redirect stderr from git hooks
Add possibility to get pure stderr output from AbortedByHookException
Change-Id: Ifc02675542dad6ced25fdd8b9fae80b5736db688
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This will change the behavior in the CLI to resemble that of C-Git more
closely by printing the stderr of the hooks to the CLI stderr
independently of the exit code of the hook.
This is also useful for the corresponding EGIT-Change, which will add
the ability to show the hook output in eclipse.
With this also the stderr can be shown even if the exit code is 0.
Bug: 553471
Change-Id: Ie7bc503fe39e270e9b93dd1108b5879f02a12b4c
Signed-off-by: Tim Neumann <Tim.Neumann@advantest.com>
This is a preparation for the corresponding EGit change to remove
redundant output in the aborted by commit hook popup.
Bug: 553469
Change-Id: Id5f39a4df659fafc9d951668e6f53fed4380de9d
Signed-off-by: Tim Neumann <Tim.Neumann@advantest.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add the following statistics
- cache hit count and hit ratio
- cache miss count and miss ratio
- count of successful and failed loads
- rate of failed loads
- load, eviction and request count
- average and total load time
Use LongAdder instead of AtomicLong to implement counters in order to
improve scalability.
Optionally expose these metrics via JMX, they are registered with the
platform MBean server if the config option jmx.WindowCacheStats = true
in the user or system level git config.
Bug: 553573
Change-Id: Ia2d5246ef69b9c2bd594a23934424bc5800774aa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.6:
Cleanup CommitAndLogCommandTest
CLI: Add support for excluding paths from Git log command
Change-Id: I2f48e16640ab1b13f0c32779422add3162260c50
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
- add missing @since tag on new field connectivityChecker
- suppress warning for this added field which may affect subclasses
which is ok
Change-Id: Id6c22258efd10fa3847290f3dcfa42893b6ae882
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Git log supports the exclude pathspec, which allows for excluding paths
from the log command. JGit only supports adding paths to the log
command. See the following StackOverflow question for details:
https://stackoverflow.com/questions/59143934/java-jgit-how-to-get-git-
commits-not-affecting-certain-directories
This commit adds an excludePath() method to the log command. It does not
yet support regex or glob wildcards.
Change-Id: I8cd59950b87850b55a15c7e2ea5470145c9aae28
Signed-off-by: John Tipper <john_tipper@hotmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Move all connectivity check to separate classes. Set default to be
FullConnectivityChecker i.e. checker which will check connectivity
from all advertised refs. Add ability to set other connectivity
checker which can use different approach.
Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: Ibb107dbfbdde8ad109be25b5ecf42be7660ef736
This method was added in stable-5.6 after 5.6 RC1 was tagged. It's shown
as an API error when using 5.6.0 RC1 as the API baseline for master.
This suppression can be removed when 5.6 is released which will serve as
the API baseline for 5.7.
Change-Id: Ic6359fe59ed2d2a9431583b0854ec86bbac76880
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The reftable format supports fast inverse (SHA1 => ref) queries.
If the ref database does not support fast inverse queries, it may be
advantageous to build a complete SHA1 to ref map in advance for
multiple uses. To let applications decide, this function indicates
whether the inverse map is available.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Idaf7e01075906972ec21332cade285289619c2b3
Same as UploadPack. These methods do not handle exceptions and the
caller can handle them. This makes it possible to customize the error
handling.
Change-Id: Ia5203f80133cb5b3e5d754859b4167ac920eaf14
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Currently, there's no way for a user to customize the error handler for
the exception happened during unpacking an incoming pack file. Create an
error handler class and let them specify one.
Change-Id: Id07638ee58c88e1365181c3ddd17ee0266f3934d
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Instead of dropping a second exception, use try-with-resources so that
exception thrown by a hook is attached as a suppressed exception.
This removes a need of holding an exception that is thrown while
unpacking an incoming packfile.
Change-Id: I74d8b944931abbf53d9d1ae5db2e889639fba027
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
BaseReceivePack#sendStatusReport anyway needs to know
CAPABILITY_REPORT_STATUS. By moving this flag to BaseReceivePack,
simplify the status reporting code.
Change-Id: Iaa0878b1fc13057b687a7f01d25c85fd78c0423e
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
This gives a chance to handle an exception for a user. For example, when
an IOException is thrown while executing
`walk.parseAny(cmd.getNewId())`, it's always handled as
REJECTED_MISSING_OBJECT. However, IOException can mean a Git storage IO
error. By introducing an error handler class, a user can add a custom
error handler for these cases.
Change-Id: I3e03a536e1d8e137cb0f6e596d71642e72adde9e
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Move exception handling code to the caller side. This is a preparation
for adding a custom error handler that is similar to UploadPack.
Note that `catch (Throwable t)` will not affect the exception spec since
Java will do an analysis of rethrown exceptions. See
https://docs.oracle.com/javase/8/docs/technotes/guides/language/catch-multiple.html.
Change-Id: I973a63d4b079c2cd1402afb3f2956e18b7d5a00c
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
This removes a raw IOException at one level. Later we'll add a custom
exception handling mechanism like UploadPack.
Change-Id: I52a7423798c97b032d848351be8b6f144776b017
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
This partially reverts Ic6bb5e66. It made private some methods in
ReceivePack that were protected before. At least two of those methods
(#init and #receivePackAndCheckConnectivity) are used by out-of-tree
subclasses for tracing and testing.
Make ReceivePack#init() and ReceivePack#receivePackAndCheckConnectivity
protected again to allow tracing and testing.
Change-Id: I7ff22c091fbfc2d94009c449d58e7c5ac9f4f759
Signed-off-by: Ivan Frade <ifrade@google.com>
When a conflicting file was blamed, JGit would not identify lines
coming from the merge parents. The main cause for this was that
Blame and BlameCommand simply added the first DirCacheEntry found
for a file to its queue of candidates (blobs or commits) to consider.
In case of a conflict this typically is the merge base commit, and
comparing a auto-merged contents against that base would yield
incorrect results.
Such cases have to be handled specially. The candidate to be
considered by the blame must use the working tree contents, but
at the same time behave like a merge commit/candidate with HEAD
and the MERGE_HEADs as parents. Canonical git does something very
similar, see [1].
Implement that and add tests.
I first did this for the JGit pgm Blame command. When I then tried
to do the same in BlameCommand, I noticed that the latter also
included some fancy but incomplete CR-LF handling. In order to
be able to use the new BlameGenerator.prepareHead() also in
BlameCommand this CR-LF handling was also moved into BlameGenerator
and corrected in doing so.
(Just considering the git config settings was not good enough,
CR-LF behavior can also be influenced by .gitattributes, and even
by whether the file in the index has CR-LF. To correctly determine
CR-LF handling for check-in one needs to do a TreeWalk with at
least a FileTreeIterator and a DirCacheIterator.)
[1] https://github.com/git/git/blob/v2.22.0/blame.c#L174
Bug: 434330
Change-Id: I9d763dd6ba478b0b6ebf9456049d6301f478ef7c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
When this class was split from DfsReftableDatabase in commit
2b1e942729 (reftable: split off generic
code from DFS code, 2019-09-26), we forgot to carry over the license
header.
Change-Id: I8698001dfb69f58784df643ef0185ab2d55e52aa
Reported-by: Masaya Suzuki <masayasuzuki@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Reftable-enabled repositories don't have a file called HEAD. Check for
reftable/ instead.
This fixes repository creation on reftable in Gerrit.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I778c2be01d96aaf135affae4b457b5fe5b483bee
Following OSGi semantic versioning which tolerates breaking implementers
in a minor release.
Change-Id: I4600c5ee9cd4ae209b69870a5d1367f83678617e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The checker is creating a new bitmap per branch leading to excessive
memory consumption. For the reachability check one bitmap with the
reachability of all branches aggregated is enough.
Build the reachability bitmap with a filter. The filter itself uses it
to emit only commits not reached before and the caller to check what
targets have been reached already.
BitmapCalculator is not required anymore.
Change-Id: Ic5c62f77fe0f188913215b7eaa51d849a9aae6a5
Signed-off-by: Ivan Frade <ifrade@google.com>
The ref points to an ObjectId that then is translated into a RevCommit.
This translation can be costly and with the incremental reachability
check is probably not needed for most of the elements.
Delay the translation from ObjectId to RevCommit to when it is needed.
Use Streams, that have the laziness built-in, all the way from Ref to
RevCommit.
This should reduce the latency for reachability checks over big sets of
references.
Change-Id: I28693087321b2beff3eaa1f3d2e7840ab0eedc6d
Signed-off-by: Ivan Frade <ifrade@google.com>
Preparatory change. Converting ObjectIds to RevCommits is potentially
expensive and in the incremental reachability check, it is probably not
required for all elements in the collection.
Pass a Stream to the reachability checker. In the follow up we make
the conversion from ObjectId to RevCommit in the stream (i.e. on
demand). This should reduce the latency of reachability checks over big
sets of references.
Change-Id: I9f310e331de5b0bf8de34143bd7dcd34316d2fba
Signed-off-by: Ivan Frade <ifrade@google.com>
UploadPack needs to check if object ids that weren't advertised before
are reachable from the references visible to the user. In the
bitmap-based reachability check, this is done incrementally: checking
against one reference, if anything remaining adding a second and so on.
It is more efficient to check first more common references (e.g. refs/heads/*)
Sort the references for the reachability checker. This should solve the
connectivity earlier and require less bitmap creation and less memory.
Change-Id: I48ac10d71e29fab2d346479802401eaea4aacb5c
Signed-off-by: Ivan Frade <ifrade@google.com>
Use lambda style comparators where possible. They are easier to read.
Change-Id: I5b80cfcd90909c94286742fa83af71015532809f
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Reftable is a binary, block-based storage format for the ref-database.
It provides several advantages over the traditional packed + loose
storage format:
* O(1) write performance, even for deletions and transactions.
* atomic updates to the ref database.
* O(log N) lookup and prefix scans
* free from restrictions imposed by the file system: it is
case-sensitive even on case-insensitive file systems, and has
no inherent limitations for directory/file conflicts
* prefix compression reduces space usage for repetitive ref names,
such as gerrit's refs/changes/xx/xxxxx format.
FileReftableDatabase is based on FileReftableStack, which does
compactions inline. This is simple, and has good median performance,
but every so often it will rewrite the entire ref database.
For testing, a FileReftableTest (mirroring RefUpdateTest) is added to
check for Reftable specific behavior. This must be done separately, as
reflogs have different semantics.
Add a reftable flavor of BatchRefUpdateTest.
Add a FileReftableStackTest to exercise compaction.
Add FileRepository#convertToReftable so existing testdata can be
reused.
CQ: 21007
Change-Id: I1837f268e91c6b446cb0155061727dbaccb714b8
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Several methods were protected when they were defined in BaseReceivePack
which has since been squashed into ReceivePack. Those methods no longer
need to be protected, and can now be private instead.
Change-Id: Ic6bb5e66f353fcbdc7919847423234af398c88b4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Prevent adding a null parent to a commit's parent array. Doing so
can cause NPEs later on.
Bug: 552160
Change-Id: Ib24b7b9b7b08e0b6f246006b4a4cade7eeb830b9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
core.quotePath = false means that "bytes higher than 0x80 are not
considered "unusal" anymore"[1], i.e., they are not escaped. In
essence this preserves non-ASCII characters in path names in output.
Note that control characters and other special characters in the
ASCII range will still be escaped.
Add a new QuotedString.GIT_PATH_MINIMAL singleton implementing this.
Change the normal GIT_PATH algorithm to use bytes instead of characters
so it can be re-used. Provide a setter in DiffFormatter for the quoting
style so that an application can override the default, which is the
setting from the git config (and by default "true"). Use the new
QuotedString.GIT_PATH_MINIMAL when core.quotePath == false.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-corequotePath
Bug: 552467
Change-Id: Ifcb233e7d10676333bf42011e32d01a4e1138059
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Some applications using JGit use their own repository caching. In
such applications, it may be needlessly inefficient to create new
submodule repositories from a SubmoduleWalk or in an IndexDiff. It
can be much more efficient to use an already cached repository
instance.
Provide a way to configure a SubmoduleWalk with a factory to create
BaseRepositoryBuilders to use to create repositories, and use it in
IndexDiff. Provide new IndexDiff.diff() operations that take such an
additional factory as parameter.
An application that caches Repository instances (for instance EGit)
can use a factory that provides builders that don't create a new
Repository instance but that return the already cached instance, if
one is available. Note that in such a case, the application may need
to be prepared to deal with IndexDiff.diff() also _closing_ the
obtained repository; if the application expects its cached Repository
instances to remain open while being cached, it'll have to use
Repository.incrementOpen() to prevent that the repository instance
gets closed.
Bug: 550878
Change-Id: Icc1b34dfc4cebd8ed4739dd09d37744d41adf711
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
IndexDiff would apply ignore mode ALL from .gitmodules to all remaining
submodules, and would ignore other settings from .gitignore and always
apply the setting defined on the IndexDiff instead. Correct that.
In canonical git the ignore setting from .gitmodules can also be
overridden by .git/config.[1] Implement that override in SubmoduleWalk.
[1] https://git-scm.com/docs/gitmodules#Documentation/gitmodules.txt-submoduleltnamegtignore
Bug: 521613
Change-Id: I9199fd447e41c7838924856dce40678370b66395
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
I1ce92869435d5eebb7d671be44561e69c6233134 merged BaseReceivePack into
ReceivePack which breaks API but is only affecting the few jgit based
servers out in the wild.
Change-Id: Iad856a2afaf3cad95d01ad81a0116cebcd9de2d9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Move the BaseReceivePack implementation back into ReceivePack. This is a
backward-incompatible change. For example, BaseReceivePack.FirstLine no
longer exists and cannot be referenced. However, most of the code
should just work by replacing BaseReceivePack with ReceivePack.
Although this is an API change, it only affects callers using JGit as a
server, and there are very few of those in the wild.
Change-Id: I1ce92869435d5eebb7d671be44561e69c6233134
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
This will provide exponential backoff with jitter to other JGit
components too.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Idd44e3bbaef6d71134ce2e3f7d405f35e7397cbd
The getPackSize method will throw IllegalStateException if it is
called when the pack size is not set. This is the case for example
when the received commands are all DELETE and there is no pack.
Add a new method hasReceivedPack that can be called prior to calling
getPackSize, to avoid causing the IllegalStateException.
See [1] for context.
[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=11918
Change-Id: I56397256a05e92c8398e65c07a859cee59b46317
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-5.5:
Fix NPE in SystemReader in tests
Change-Id: I8500caeca99fe05ce9a80b54330b72e23670a46b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
SystemReader.updateAll() must _not_ test whether the file exists. In
tests at least there are FileBasedConfigs with a null file. Test
configs should (and do) override isOutdated() to deal with this case.
Change-Id: I56303fe0d56afeb9f2203ee807a92c5dcf3809e9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.5:
BaseReceivePack: Fix the format
Prepend hostname to subsection used to store file timestamp resolution
Store filesystem timestamp resolution in extra jgit config
SystemReader: extract updating config and its parents if outdated
Change-Id: Iecfddce8081303af29badcdcd3d72a0da50c964f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This ensures the measured filesystem timestamp resolution will be only
used on the machine where it was measured and avoid errors in case the
~/.jgitconfig file is copied to another machine.
Bug: 551850
Change-Id: Iff2a11be62ca94c3bbe4a955182988dc50852f9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This avoids polluting hand-crafted user level config with
auto-configured options which might disturb in environments where
the user level config is replicated between different machines.
Add a jgit config as parent of the system level config. Persist
measured timestamp resolutions always in this jgit config and read it
via the user global config. This has the effect that auto-configured
timestamp resolution will be used by default and can be overridden in
either the system level or user level config.
Store the jgit config under the XDG_CONFIG_HOME directory following the
XDG base directory specification [1] in order to ensure that we have
write permissions to persist the file. This has the effect that each OS
user will use its jgit config since they typically use different
XDG_CONFIG_HOME directories.
If the environment variable XDG_CONFIG_HOME is defined the jgit config
file is located at $XDG_CONFIG_HOME/jgit/config otherwise the default is
~/.config/jgit/config.
If you want to avoid redundant measurement for different OS users
manually copy the values measured and auto-configured for one OS user to
the system level git config.
[1] https://wiki.archlinux.org/index.php/XDG_Base_Directory
Bug: 551850
Change-Id: I0022bd40ae62f82e5b964c2ea25822eb55d94687
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Change-Id: Ia77f442e47c5670c2d6d279ba862044016aabd86
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
- filter errors for new APIs added in service release
- remove unused filters
Change-Id: Ifbf532b8a3c46d4ed78a38f6c75073a072b7f669
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Pass the CheckoutMetadata to checkoutEntry(), otherwise cr-lf settings
and smudge filters would be ignored.
Change-Id: Ifb1c4cb098a0b47c9752d0231d530db6a92b25a4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
A merge may write files to the working tree. After a successful
merge one must fire a WorkingTreeModifiedEvent explicitly if
getModifiedFiles() is not empty.
Also, any touched files must be reported by the
WorkingTreeModifiedEvent fired by DirCacheCheckout.checkout().
Bug: 552636
Change-Id: I5fab8279ed8be8a4ae34cddfa726836b9277aea6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
MergedReftableTest#scanDuplicates tests whether we can write duplicate
keys in a merged reftable. Apparently, the first key appearing should
get precedence, and this works because the sort() algorithm on ordered
collections is stable.
This is potentially confusing behavior, because you can write data
into the table that cannot be retrieved (Merged table can only have
one entry per key), and the APIs such as exactRef() only return a
single value.
Make this consistent with behavior introduced in I04f55c481 "reftable:
enforce ordering for ref and log writes" by considering a duplicate key
in sortAndWriteRefs as a fatal runtime error.
Change-Id: I1eedd18f028180069f78c5c467169dcfe1521157
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Some URLs cannot be converted via URL.toURI(). So don't convert
the full URL but only the bits that are needed to find a proxy
via java.net.ProxySelector.
Bug: 549690
Change-Id: I55b5ecee70c6b52f72f9bdba9ce552fde7f33976
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The RepositoryState's description is shown in EGit's interactive rebase
view. Use the grammatically correct form.
Bug: 552359
Change-Id: I860ef730a67f2b34d7fabbfbda5ea65ead9270d2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Instead log the problem. There is no reason to panic when we fail to
delete a single temporary file in java.io.tmpdir.
Change-Id: Idb867b3f07b090c7453ccd3688e94097df3b2672
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This fixes OperatorPrecedence error raised by errorprone on FS#findHook.
Change-Id: Ia15f61902c7deff7328c1afa066fc53152949bbf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Instead of just looking for a substring match of user.signingKey
in a key's user ID implement the GPG matching formats[1] for:
'=' Full exact match
'<' Full exact match of the e-mail address
'@' Substring match within the e-mail address only
'*' General case-insensitive substring match (default)
When user.signingKey is not set, the committer's e-mail address is
used by default. In that case, use '<', i.e., require an exact match
on the OpenPGP e-mail address.
Also handle the optional "0x" prefix for (partial) key fingerprints.
[1] https://www.gnupg.org/documentation/manuals/gnupg/Specify-a-User-ID.html
Bug: 550335
Change-Id: I6ce482a099ff1a0dc9de45435cd4d3ec5b504f12
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The effect of assert is defined by compiler flags, so this code
introduced a potential vector for corruption.
Change-Id: I12197432e4351a5bd4aa24d352a19937721845c3
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Currently when a GPG key is looked up using a user identity the first
key from the keyring that has this user identity is returned.
The code was changed to instead return the first signing [S] key in this
keyring and only return the master key if no such signing key was found.
If the master key also does not have the signing flag set null is
returned instead.
Bug: 552288
Change-Id: I194862991d13c2c7ff34a60a54a227167f88f53b
Signed-off-by: Roan Hofland <roan.hofland@hotmail.com>
Currently when a subkey is configured for signing via the git
user.signingkey configuration option the first key from the keyring for
this subkey would be returned for use (master key). The code has been
changed to return the requested key from the keyring instead.
Bug: 552288
Change-Id: I1c1cdf64c1667316a274ff9d829fc2b563797f2a
Signed-off-by: Roan Hofland <roan.hofland@hotmail.com>
Support the core.hooksPath git config. This can be an absolute or
relative path of a directory where to find git hooks; a relative
path is resolved relative to the directory the hook will run in.
Bug: 500266
Change-Id: I671999a6386a837e897c31718583c91d8035f3ba
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Since [1] the gerrit project includes jgit as a submodule, and has this
warning enabled, resulting in 100s of warnings in the console.
Also enable the warning here, and fix them.
At the same time, add missing braces around adjacent and nearby one-line
blocks.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/227897
Change-Id: I81df3fc7ed6eedf6874ce1a3bedfa727a1897e4c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This introduces ReftableBatchRefUpdate and ReftableDatabase, as
generic classes, with some code moved to DfsReftableBatchRefUpdate and
DfsReftableDatabase.
Clarify thread-safety requirements by asserting locked status in
accessors, and acquiring locks in callers. This does not fix threading
problems, because ReftableBatchRefUpdate already wraps the whole
transaction in a lock.
This also fixes a number of bugs in ReftableBatchRefUpdate:
* non-atomic updates should not bail on first failure
* isNameConflicting should also check for conflicts between names that
are added and removed in the BatchRefUpdate.
Change-Id: I5ec91173ea9a0aa19da444c8c0b2e0f4e8f88798
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
On changing a ref, the old SHA1 is not updated in the object => ref
mapping. This means search by object ID may still turn up a ref from
deeper within the stack. To fix this, check all refs produced by the
merged iterator against the merged reftables.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I41e9cd395b0608eedeeaead0a9fd997238d747c9
The flag enabling sideband-all is used in two places: in UploadPack
for advertisement and in the protocol parser to read it from the
request.
This leds to problems in distributed deployments where the two requests of
a fetch can go to different servers with different configurations.
Use the existing allowsidebandall to accept the sideband-all request
(and respond to it) and introduce a new "advertisesidebandall" to toggle
the advertising of the feature.
Change-Id: I892d541bc3f321606c89bad1d333b079dce6b5fa
Signed-off-by: Ivan Frade <ifrade@google.com>
If the SubModuleWalk isn't closed its TreeWalk's ObjectReader won't
be closed. Re-loading the DirCache during an IndexDiff is not only
inefficient but could also give strange results if an external
process had modified the index in the meantime: file diffs would
be based on a "before" state, but submodule diffs on an "after"
state.
Change-Id: Iab948c08ac342138b37263c9028d80b84101f6d6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Add to statistics the amount and size of packfiles offloaded to HTTP
download.
Change-Id: I895a7219ecac2794368bfc4fdfae74c1238deed9
Signed-off-by: Ivan Frade <ifrade@google.com>
The object identifying packfiles to send them via packfile-uri contains
only the uri and the hash. This is the information that goes through the
wire. It would be useful to know also the size of those packfile, for
example to track how many bytes have been offloaded to HTTP.
Add size field the CachedPackUriProvider.PackInfo object.
Change-Id: If6b921b48a4764d936141c777879b148cc80bbd3
Signed-off-by: Ivan Frade <ifrade@google.com>
* changes:
UploadPack: Create a method that propagates an exception as-is
UploadPack: Consolidate the sideband handling code to one place
UploadPack: Introduce ErrorWriter
Exception handling can be isolated from UploadPack. This makes it
possible to make the exception handler pluggable.
Change-Id: Ieebbd6711963c7f2e47a98783b4ad815793721c7
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
This consolidates the sideband stream creation code and the error
handling code for the sideband-allowed part in the Git protocol to one
place.
Change-Id: I0e3e94564f50d1be32006f9d8bcd1ef1ce6bf07e
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
ErrorWriter writes an error message to the user. The implementation is
swapped once it detects that the client supports sideband. By default it
uses the protocol level ERR packet, which was introduced recently.
In total the error output is done in two different places;
UploadPack#upload and UploadPack#sendPack. These will be consolidated in
the next change.
Change-Id: Ia8d72e31170bbeafc8ffa8ddb92702196af8a587
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Also correctly parse the "always" value (allowed in canonical git
since git 2.12.0[1]). Adapt the ReflogWriter.
[1] https://github.com/git/git/commit/341fb2862
Bug: 551664
Change-Id: I051c76ca355a2ac8d6092de65f44b18bf9aeb125
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.5:
Prepare 5.5.2-SNAPSHOT builds
JGit v5.5.1.201910021850-r
Prepare 5.3.7-SNAPSHOT builds
JGit v5.3.6.201910020505-r
Prepare 5.1.13-SNAPSHOT builds
JGit v5.1.12.201910011832-r
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
Change-Id: Iabebdc4786b52ec678caa798975428fda4ad7bd7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.4:
Prepare 5.3.7-SNAPSHOT builds
JGit v5.3.6.201910020505-r
Prepare 5.1.13-SNAPSHOT builds
JGit v5.1.12.201910011832-r
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
Change-Id: Ia1070fd170651ce827bc6b876c6764a44ffe60eb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.3:
Prepare 5.3.7-SNAPSHOT builds
JGit v5.3.6.201910020505-r
Prepare 5.1.13-SNAPSHOT builds
JGit v5.1.12.201910011832-r
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
Change-Id: Ia621d06a9489ee276c793de9dd4a77f4ff19e2ac
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.2:
Prepare 5.1.13-SNAPSHOT builds
JGit v5.1.12.201910011832-r
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
Change-Id: I3f11a83e177daefa0a2e91173f70f9547067f713
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
# By Matthias Sohn (2) and Saša Živkov (1)
* stable-5.1:
Prepare 5.1.13-SNAPSHOT builds
JGit v5.1.12.201910011832-r
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
Change-Id: Iaae4b171eaa0081f9142489de8df94ab455d65f7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
MergedReftable is not used as an AutoCloseable, because closing tables
is currently handled by DfsReftableStack#close.
Encode that a MergedReftable is a list of ReftableReaders. The previous
code suggested that we could form nested trees of MergedReftables,
which is not how we use reftables.
Change-Id: Icbe2fee8a5a12373f45fc5f97d8b1a2b14231c96
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
In the Config#StringReader we relied on ArrayIndexOutOfBoundsException
to detect the end of the input. Creation of exception with (deep) stack
trace can significantly degrade performance in case when we read
thousands of config files, like in the case when Gerrit reads all
external ids from the NoteDb.
Use the buf.length to detect the end of the input.
Change-Id: I12266f25751373a870ce3fa623cf2a95d882d521
Otherwise the paths modified by a cherry-pick with conflicts won't be
reported as modified via WorkingTreeModifiedEvents.
Change-Id: I875b67c0d2f68efdf90a9c32b80a2e074ed3570d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
This makes the intended use of the classes more clear. It also
simplifies generic functions that write reftables: they only need a
ReftableWriter as argument, as the stream is carried within the
ReftableWriter.
Change-Id: Idbb06f89ae33100f0c0b562cc38e5b3b026d5181
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The functionality in ReftableStack is specific to DFS.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: If6003d104b1ecb0f3ca7e9c3815b233fa0abf077
This makes maxUpdateIndex() available in MergedReftable, so we can
know generically at which index to create the next reftable in a
stack.
Change-Id: Ia2314bc57c8b5dd7e69d5e61096fdce1d35abd11
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Older JGit stored only milliseconds timestamps in the index. Newer
JGit may get finer timestamps from the file system. This leads to
slow index diffs when a new JGit runs against an index produced
by older JGit because many timestamps will differ and JGit will
then do many content checks. See [1].
Handle this migration case by only comparing milliseconds if the
index entry has only millisecond precision.
The inverse may also occur; also compare only milliseconds if the
file timestamp has only millisecond precision.
Do the same also for microsecond resolution. On Windows, NTFS may
provide 100ns resolution and may be used by external programs writing
the index, but Java's WindowsFileAttributes may provide only
microseconds.
File timestamp precision in Java depends not only on the Java APIs
used by different JGit versions but may also change when running the
same Java code on different VMs. And of course the resolution may
vary among operating and file systems. Moreover, timestamp precision
in the index depends on the program that wrote the index. Canonical
git may use a different resolution, maybe even different between git
versions.
[1] https://www.eclipse.org/forums/index.php/t/1100344/
Change-Id: Idfd08606c883cb98787b2138f9baf0cc89a57b56
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The removed code was trying to avoid mistakenly reporting differences
when core.autocrlf was set to "input" but a file had already been
committed with CR-LF. It did that by running the blob from the cache
through a CRLF-to-LF filter because older JGit would also run the file
from the working tree through such a filter.
The real fix for this case was done in commit 60cf85a. Since then files
are not normalized if they have already been committed with CR-LF and
this old fix attempt from bug 372834 is no longer needed.
Change-Id: Ib4facc153d81325cb48b4ee956a596b423f36241
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
This removes one use of DFS specific code in this class.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I3ef6a4b98357cc6dc480892244ddc51d2fd751a2
This lets us write reftables generically with functions that take
just ReftableWriter argument
Change-Id: I7285951f62f9bd4c78e8f0de194c077d51fa4e51
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
allRefs determined the end of the ref block without accounting for
index or log blocks. This could cause other blocks to be interpreted
as ref blocks, leading to "invalid block" error messages.
Change-Id: I7b9323e7d5e0e7d64535b3ec1efd576aed1e9870
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
If CheckStat is MINIMAL or timestamps have no nanosecond part
WorkingTreeIterator.compareMetaData only checks the second part of
timestamps and ignores nanoseconds which may have ended up in the index
by using native git.
If
fileLastModified.getEpochSecond() == cacheLastModified.getEpochSecond()
we currently proceed comparing fileLastModified and cacheLastModified
with full precision which is wrong since we determined that we detected
reduced timestamp resolution.
Fix this and also handle smudged index entries for CheckStat.MINIMAL.
Change-Id: I6149885903ac63d79b42d234cc02aa4e19578f3c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.5:
bazel: fix running http tests
Set parameter name in parameterized http tests
Format BUILD files with buildifier
Format BUILD files with buildifier
[error prone] Suppress NonAtomicVolatileUpdate in SimpleLruCache
Bazel: Format BUILD files with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: Idb0848dd33a76328b24908dc86db335cca742a1c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.4:
Format BUILD files with buildifier
Format BUILD files with buildifier
[error prone] Suppress NonAtomicVolatileUpdate in SimpleLruCache
Bazel: Format BUILD files with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: Idf3abd80ad3b00188f655e638d9908228770911f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.3:
Format BUILD files with buildifier
[error prone] Suppress NonAtomicVolatileUpdate in SimpleLruCache
Bazel: Format BUILD files with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: Iaea77a89fcd821df5dbb24f5e3e39e6d1dfc0bd4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.2:
[error prone] Suppress NonAtomicVolatileUpdate in SimpleLruCache
Bazel: Format BUILD files with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: I0d8b7f79177a20dc00c89e2cf0005eb3d3039532
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
We don't need to update time atomically since it's only used to order
cache entries in LRU order.
Change-Id: I756fa6d90b180c519bf52925f134763744f2c1f1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.1:
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: Iea37a8e39e9d4872dc607c9222dcf191ce4e4757
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Since commit c3b0dec509fe136c5417422f31898b5a4e2d5e02, Git has
disallowed writing a non-commit to refs/heads/* refs. JGit still
allows that, which can put users in a bad situation. For example,
git push origin v1.0:master
pushes the tag object v1.0 to refs/heads/master, instead of the
intended commit object v1.0^{commit}.
Prevent that by validating that the target of the ref points to a
commit object when pushing to refs/heads/*.
Git performs the same check at a lower level (in the RefDatabase). We
could do the same here, but for now let's start conservatively by
handling it in pushes first.
[jn: fleshed out commit message]
Change-Id: I8f98ae6d8acbcd5ef7553ec732bc096cb6eb7c4e
Signed-off-by: Yunjie Li <yunjieli@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
* stable-5.2:
Prepare 5.1.12-SNAPSHOT builds
JGit v5.1.11.201909031202-r
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: I7e88c7c7d202f1e3fb8e143277650aa5fefff439
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.1:
Prepare 5.1.12-SNAPSHOT builds
JGit v5.1.11.201909031202-r
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Iac5dc9683cea97db04d20f27c10f2e103d3ae7b5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-5.0:
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: I172136a031ff0730e575327cafb3527c9650a71d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.11:
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Ifb6a4dbea2f48fd2ffa66eb737d61920aefedfbd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.10:
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: If672b4f0c350f4e8ff7e1e706485cffd8137236d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.9:
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Id5bf44645655fca40ad22bb1f1ad20a7c2e8f6db
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
PackedBatchRefUpdate was creating a new packed-refs list that was
potentially unsorted. This would be papered over when the list was
read back from disk in parsePackedRef, which detects unsorted ref
lists on reading, and sorts them. However, the BatchRefUpdate also
installed the new (unsorted) list in-memory in
RefDirectory#packedRefs.
With the timestamp granularity code committed to stable-5.1, we can
more often accurately decide that the packed-refs file is clean, and
will return the erroneous unsorted data more often. Unluckily timed
delays also cause the file to be clean, hence this problem was
exacerbated under load.
The symptom is that refs added by a BatchRefUpdate would stop being
visible directly after they were added. In particular, the Gerrit
integration tests uses BatchRefUpdate in its setup for creating the
Admin group, and then tries to read it out directly afterward.
The tests recreates one failure case. A better approach would be to
revise RefList.Builder, so it detects out-of-order lists and
automatically sorts them.
Fixes https://bugs.eclipse.org/bugs/show_bug.cgi?id=548716 and
https://bugs.chromium.org/p/gerrit/issues/detail?id=11373.
Bug: 548716
Change-Id: I613c8059964513ce2370543620725b540b3cb6d1
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add the constant, and implement hashing of known host names in
OpenSshServerKeyDatabase. Add a test verifying that the hashing
works.
Bug: 548492
Change-Id: Iabe82b666da627bd7f4d82519a366d166aa9ddd4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* changes:
dfs: Take size as long instead of int
dfs: Read at the aligned position
dfs: Add a position argument
dfs: Move the deeply nested code to its own method
* stable-5.5:
Return a new instance from openSystemConfig and openUserConfig
Change-Id: I3ba9fa8ceced6e9f883fc23afb0e2c812391beac
Signed-off-by: Jonathan Nieder <jrn@google.com>
* stable-5.4:
Return a new instance from openSystemConfig and openUserConfig
Change-Id: If576513ebbd80c7f01b83be5ab2edd436f5739e3
Signed-off-by: Jonathan Nieder <jrn@google.com>
* stable-5.3:
Return a new instance from openSystemConfig and openUserConfig
Change-Id: I5f2f957ff27b7941ab29b2dc1c885fbf0093705a
Signed-off-by: Jonathan Nieder <jrn@google.com>
* stable-5.2:
Return a new instance from openSystemConfig and openUserConfig
Change-Id: I6491549ab13aad2a4c3f8444a090a94a378eccdb
Signed-off-by: Jonathan Nieder <jrn@google.com>
* stable-5.1:
Return a new instance from openSystemConfig and openUserConfig
Change-Id: I93afa16b14e3e2d127705e3378fb703d6c329694
Signed-off-by: Jonathan Nieder <jrn@google.com>
Move the handling of cached user and system config to getSystemConfig
and getUserConfig methods and revert the implementation of
openSystemConfig and openUserConfig to the old stateless
implementation.
This ensures the open methods respect the passed-in parent config, which
may be different on each invocation. Additionally, returning a new
instance matches the behavior of the previous implementation of the
default system reader, which downstream callers may be depending on.
Move the implementation of the new caching methods getSystemConfig and
getUserConfig up to SystemReader. This avoids that we break the ABI for
subclasses of SystemReader.
Also see [1] which fixed a similar problem with Gerrit's custom
SystemReader.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/225458
Change-Id: If54a2491932d8fc914d4649cb73c9e837c5b8ad0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
These warnings were missed to address in a0048208 which introduced them.
Change-Id: Ia2d15fdce72c10378d020682b80fe7fc548c0d4c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>