We've used "GIT_EDITOR=edit git config --system --edit" to determine
the location of the git system config for a long time. But git 2.34.0
always expects this command to have a tty, but there isn't one when
called from Java. If there isn't one, the Java process may get a
SIGTTOU from the child process and hangs.
Arguably it's a bug in C git 2.34.0 to unconditionally assume there
was a tty. But JGit needs a fix *now*, otherwise any application using
JGit will lock up if git 2.34.0 is installed on the machine.
Therefore, use a different approach if the C git found is 2.8.0 or
newer: parse the output of
git config --system --show-origin --list -z
"--show-origin" exists since git 2.8.0; it prefixes the values with
the file name of the config file they come from, which is the system
config file for this command. (This works even if the first item in
the system config is an include.)
Bug: 577358
Change-Id: I3ef170ed3f488f63c3501468303119319b03575d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
(cherry picked from commit 9446e62733)
- jetty 9.4.30 moved to archive.eclipse.org
- use the final release p2 repo instead of staging for 2020-09
Change-Id: Ia096200e5983f0022c6c0da4dae035433e852807
We observed the following issue:
$ git tag -a v1.0 -m v1.0
$ git push origin tag v1.0
$ git ls-remote origin v1.0^{}
... empty result ...
On the server (Gerrit) side run git-gc to pack the refs:
$ git gc
Repeat the ls-remote from the client and the result is correct:
$ git ls-remote origin v1.0^{}
7ad85c810de3ae922903d4bdd17c53cd627260ba refs/tags/v1.0^{}
Unfortunately, the existing UploadPackTest didn't reveal this issue
although it provided the test case needed to do so: testV2LsRefsPeel.
This is because The UploadPackTest uses InMemoryRepository which
internally uses Dfs* implementations. The issue is only reproducible
when using the FileRepository.
It is a non-trivial task to refactor the UploadPackTest to work against
both InMemoryRepository and FileRepository and this change is not trying
to do that. This change creates a new test:
UploadPackLsRefsFileRepositoryTest and copies the necesssary code from
the UploadPackTest.
Change-Id: Icfc7d0ca63f1524bafe24c9626ce12ea72aa3718
Signed-off-by: Saša Živkov <sasa.zivkov@sap.com>
* stable-5.8:
reftable: drop code for truncated reads
reftable: pass on invalid object ID in conversion
Update eclipse-jarsigner-plugin to 1.3.2
Change-Id: I9e66ef90dc9a65bac47b35705d679bf992bd72b9
* stable-5.7:
reftable: drop code for truncated reads
reftable: pass on invalid object ID in conversion
Update eclipse-jarsigner-plugin to 1.3.2
Change-Id: I88c47ff57f4829baec5b19aad3d8d6bd21f31a86
* stable-5.6:
reftable: drop code for truncated reads
reftable: pass on invalid object ID in conversion
Update eclipse-jarsigner-plugin to 1.3.2
Change-Id: I1c18f5f435f4a4a86e0548a310dbfc74191e1ed5
The reftable format is a block based format, but allows for variably
sized blocks. This obviously happens for reflog blocks (which are zlib
compressed), but is also accepted for index blocks: In the spec, this
is motivated as
To achieve constant O(1) disk seeks for lookups the index must be
a single level, which is permitted to exceed the file's
configured block size, but not the format's max block size of
15.99 MiB.
Hence, when parsing a block, one cannot be sure of its exact size:
after reading a default-size block (eg. 4kb), the block header may
state that the block is in fact larger.
Before, the code would mark the block as `truncated`, noting
// Its OK during sequential scan for an index block to have been
// partially read and be truncated in-memory. This happens when
// the index block is larger than the file's blockSize. Caller
// will break out of its scan loop once it sees the blockType.
This looks like either
* a remnant of never-implemented functionality. There is no reason to
ever sequentially scan an index block.
* alluding to sequential scan of the data blocks before the index
blocks (eg. scanning refs, which ends when we find the first ref index
block, and we can then ignore the index block).
This comment is followed by code that populates the
restartTbl/restartCnt fields relative to the (possibly truncated)
buffer. If the buffer is truncated, this essentially reads garbage,
leading to OOB array access when using the index block.
Fix this by dropping the truncated logic and issuing a second read if
the first read was short.
Add a test.
We have never observed this failure scenario at Google. We use 64kb
blocksize, which requires us to need fewer index entries. The reftable
spec mentions an Android repo of size 36M. With 64kb blocks, that's
just 562 index entries. Even with historical growth, we are long from
requiring an index whose size exceeds a single block.
When adding the analogous test for seeking refs, there was no failure.
This points to another possibility which is that the code tries to
avoid writing large index blocks for refs.
I did not investigate further which one it is.
Fixes https://bugs.eclipse.org/bugs/show_bug.cgi?id=576250
Bug: 576250
Change-Id: I41ec21fac9e526ef57b3d6fb57b988bd353ee338
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Before, while trying to determine if an object ID was a tag or not,
the reftable conversion would yield an exception.
Change-Id: I3688a0ffa9e774ba27f320e3840ff8cada21ecf0
GitServlet delegates repository access over HTTP to the GitFilter
servlet.
GitServlet, in turn, can be extended by jgit consumers to provide custom
logic when handling such operations.
This is the case, for example, with Gerrit Code Review, which provides
custom behavior with a GitOverHttpServlet [1].
Among possible customizations, the ability of specifying a custom error
handler for UploadPack and ReceivePack was already introduced in
GitFilter by Idd3b87d6b and I9c708aa5a2, respectively.
However the `setUploadPackErrorHandler` and `setReceivePackErrorHandler`
methods were never added to the GitServlet.
Expose the `setUploadPackErrorHandler` and `setReceivePackErrorHandler`
methods to the GitServlet, so that consumers of the jgit library might
specify custom error handlers.
[1] https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-3.2/java/com/google/gerrit/httpd/GitOverHttpServlet.java#95
Change-Id: I712d485ff68b662b48c71ef75650c5a155950d23
* stable-5.8:
[test] Create keystore with the keytool of the running JDK
ReachabilityCheckerTestCase: fix reachable from self test case
Change-Id: If09cbb877c674f15261715cecef7a2393bf66fa3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.7:
[test] Create keystore with the keytool of the running JDK
ReachabilityCheckerTestCase: fix reachable from self test case
Change-Id: I32010c6bf45d5138e17143d6c284ac56434eade1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.6:
[test] Create keystore with the keytool of the running JDK
ReachabilityCheckerTestCase: fix reachable from self test case
Change-Id: I1f6b4fc26f6ee6f22cc0aacd032c1e73ba246dbc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.5:
[test] Create keystore with the keytool of the running JDK
Change-Id: Icb0bb6dc4ad05b1f3eb562547893f2e0aedf8775
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.4:
[test] Create keystore with the keytool of the running JDK
Change-Id: I5ff3dc1c771aeb33af39eb68f166c7282b478cf8
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.3:
[test] Create keystore with the keytool of the running JDK
Change-Id: If92372b7bfbfb9445fcb934c48dc1cd6610e061b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.2:
[test] Create keystore with the keytool of the running JDK
Change-Id: I981de862c614986a7b443fed1cce7b895b758682
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-5.1:
[test] Create keystore with the keytool of the running JDK
Change-Id: Ic56f4f23c37432377be88e07d06c5ad75591d84f
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Call keytool with the absolute path of "java.home". Otherwise a keytool
for a different, maybe even newer Java version might be picked up, and
then the keystore may not be readable by the JVM used to run the tests.
(cherry picked from commit 2d73c702d3)
Change-Id: Iea77024947a34267f008847d81312fe0abadc615
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
When reading loose objects over NFS it is possible that the OS syscall
would fail with ESTALE errors: This happens when the open file
descriptor no longer refers to a valid file.
Notoriously it is possible to hit this scenario when git data is shared
among multiple clients, for example by multiple gerrit instances in HA.
If one of the two clients performs a GC operation that would cause the
packing and then the pruning of loose objects, the other client might
still hold a reference to those objects, which would cause an exception
to bubble up the stack.
The Linux NFS FAQ[1] (at point A.10), suggests that the proper way to
handle such ESTALE scenarios is to:
"[...] close the file or directory where the error occurred, and reopen
it so the NFS client can resolve the pathname again and retrieve the new
file handle."
In case of a stale file handle exception, we now attempt to read the
loose object again (up to 5 times), until we either succeed or encounter
a FileNotFoundException, in which case the search can continue to
Packfiles and alternates.
The limit of 5 provides an arbitrary upper bounds that is consistent to
the one chosen when handling stale file handles for packed-refs
files (see [2] for context).
[1] http://nfs.sourceforge.net/
[2] https://git.eclipse.org/r/c/jgit/jgit/+/54350
Bug: 573791
Change-Id: I9950002f772bbd8afeb9c6108391923be9d0ef51
Rather than getting all ref names and prefixes and saving them
in memory to perform the check for conflicting names, rely on
RefDirectory.isNameConflicting as it is no longer an expensive
call after it was optimized in Ie994fc.
The old optimization to save ref names and prefixes in memory
was targeted towards making clones faster. With this change,
the clone performance is unaffected when tests were done with
repos containing many(~500k) refs.
Here are few recorded elapsed times for creating 10 branches
using BatchRefUpdate on NFS based repositories with varying
loose refs count. As seen here, this change helps improve the
BatchRefUpdate performance from O(n^2) to O(1).
loose_refs_count with_change without_change
50 241 ms 310 ms
300 263 ms 1502 ms
1k 181 ms 4241 ms
2k 204 ms 6440 ms
9k 158 ms 25930 ms
20k 154 ms 60443 ms
50k 171 ms 135199 ms
110k 157 ms 329450 ms
160k 209 ms 396328 ms
This update improves the Gerrit notedb migration performance
as it uses BatchRefUpdate to write change meta refs similar to
the test performed above.
Change-Id: I853ac6c7feb4b39c3156c01876b38cbd182accfe
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
Update tests to record the number of events fired post-setup and only
assert for events fired during BatchRefUpdate.execute. For tests which
use writeLooseRef to setup refs, create new tests which assert the
number of RefsChangedEvent(s) rather than updating the existing ones
to call RefDirectory.exactRef as it changes the code path.
Change-Id: I0187811628d179d9c7e874c9bb8a7ddb44dd9df4
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
Avoid having to scan over ALL loose refs to determine if the
name is nested within or is a container of an existing reference.
This can get really expensive if there are too many loose refs.
Instead use exactRef and getRefsByPrefix which scan based on a
prefix.
With a simple shell script(like below) using jgit client to create
1k refs in a new repository on NFS, this change brings down the time
from 12mins to 7mins.
for ref in $(seq 1 1000); do
jgit branch "$ref"
done
Here are few recorded elapsed times to create a new branch on NFS
based repositories with varying loose refs count. As we see here,
this change improves the name conflicting check from O(n^2) to O(1).
loose_refs_count with_change without_change
50 44 ms 164 ms
300 45 ms 1193 ms
1k 38 ms 2610 ms
2k 44 ms 6003 ms
9k 46 ms 27860 ms
20k 45 ms 48591 ms
50k 51 ms 135471 ms
110k 43 ms 294252 ms
160k 52 ms 430976 ms
Change-Id: Ie994fc184b8f82811bfb37b111eb9733dbe3e6e0
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
- bazlets need to be updated to react on Maven central no longer
supporting http protocol but only https
- update bazel to 2.0
Change-Id: I07f5f050f3b1db2014a5198a28b6bbf893434814
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit d74daad1e0)
* stable-5.8:
Remove texts which were added by mistake in 00386272
Fix formatting which was broken in 00386272
Change-Id: I9ca7a0237f87d1d4bcaba81e709eaa67902f27e5
* stable-5.7:
Remove texts which were added by mistake in 00386272
Fix formatting which was broken in 00386272
Change-Id: I7ed3f47cb46e6c1bf483702c8925a24e88658e47
* stable-5.6:
Remove texts which were added by mistake in 00386272
Fix formatting which was broken in 00386272
Change-Id: I45d444b360485564744bf3dfad2c2f5a5e7fcdf6
Don't create the stream eagerly in lock(); that may cause JGit to
exceed OS or JVM limits on open file descriptors if many locks need
to be created, for instance when creating many refs. Instead create
the output stream only when one really needs to write something.
Bug: 573328
Change-Id: If9441ed40494d46f594a896d34a5c4f56f91ebf4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
In a distributed setting, one can have multiple datacenters use
reftables for serving, while the ground truth for the Ref database is
administered centrally. In this setting, replication delays combined
with compaction can cause update-index ranges to overlap.
Such a setting is used at Google, and the JGit code already handles
this correctly (modulo a bugfix that applied in change I8f8215b99a).
Remove the restriction that was applied at FileReftableDatabase.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I6f9ed0fbd7fbc5220083ab808b22a909215f13a9
Thomas Wolf