Commit Graph

8454 Commits

Author SHA1 Message Date
Alina Djamankulova 3cd7eb1b23 DFS block cache: Refactor to enable parallel index loading
With this change bitmap index creation is not blocked on index and
reverse index full initialization in DfsPackFile. Now bitmap index and index could be
read from storage in parallel in separate threads.

Update PackBitmapIndexV1 constructor to get packIndex and reverseIndex
suppliers instead of actual objects. Inner class IdxPositionBitmap was added to initialize pack objects after bitmap
index is fully read from storage. Update DfsPackFile and PackBitmapIndex
accordingly.

Signed-off-by: Alina Djamankulova <adjama@google.com>
Change-Id: Iea59ab58501b2acbbf9263412982ec9c6898a7ee
2021-08-12 14:51:18 -04:00
Thomas Wolf 1825a2230c Merge branch 'stable-5.12'
* stable-5.12:
  [test] Create keystore with the keytool of the running JDK
  Fix garbage collection failing to delete pack file
  ReachabilityCheckerTestCase: fix reachable from self test case

Change-Id: I4b54f4850819736144edb784617ee902f491ffd6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 18:19:23 +02:00
Thomas Wolf 65de0d4863 Merge branch 'stable-5.11' into stable-5.12
* stable-5.11:
  [test] Create keystore with the keytool of the running JDK
  ReachabilityCheckerTestCase: fix reachable from self test case

Change-Id: Ie8db450a1fad05bddb812a55b2ceb03b2805403a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:12:13 +02:00
Thomas Wolf 1a963b3976 Merge branch 'stable-5.10' into stable-5.11
* stable-5.10:
  [test] Create keystore with the keytool of the running JDK
  ReachabilityCheckerTestCase: fix reachable from self test case

Change-Id: I55f4dd19e9fa3b789bd9a79d256fe9abb55ee7f4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:11:37 +02:00
Thomas Wolf bae2f86ac7 Merge branch 'stable-5.9' into stable-5.10
* stable-5.9:
  [test] Create keystore with the keytool of the running JDK
  ReachabilityCheckerTestCase: fix reachable from self test case

Change-Id: Ic37426211905d987ddd11480a54d95b86143c94c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:10:56 +02:00
Thomas Wolf 59aec9b15a Merge branch 'stable-5.8' into stable-5.9
* stable-5.8:
  [test] Create keystore with the keytool of the running JDK
  ReachabilityCheckerTestCase: fix reachable from self test case

Change-Id: If09cbb877c674f15261715cecef7a2393bf66fa3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:10:16 +02:00
Thomas Wolf 60bffc2fa8 Merge branch 'stable-5.7' into stable-5.8
* stable-5.7:
  [test] Create keystore with the keytool of the running JDK
  ReachabilityCheckerTestCase: fix reachable from self test case

Change-Id: I32010c6bf45d5138e17143d6c284ac56434eade1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:09:39 +02:00
Thomas Wolf ae56213210 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  [test] Create keystore with the keytool of the running JDK
  ReachabilityCheckerTestCase: fix reachable from self test case

Change-Id: I1f6b4fc26f6ee6f22cc0aacd032c1e73ba246dbc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:09:00 +02:00
Thomas Wolf 5bf6cfe32b Merge branch 'stable-5.5' into stable-5.6
* stable-5.5:
  [test] Create keystore with the keytool of the running JDK

Change-Id: Icb0bb6dc4ad05b1f3eb562547893f2e0aedf8775
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:07:25 +02:00
Thomas Wolf d6fab62f9e Merge branch 'stable-5.4' into stable-5.5
* stable-5.4:
  [test] Create keystore with the keytool of the running JDK

Change-Id: I5ff3dc1c771aeb33af39eb68f166c7282b478cf8
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:06:25 +02:00
Thomas Wolf e264a64190 Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  [test] Create keystore with the keytool of the running JDK

Change-Id: If92372b7bfbfb9445fcb934c48dc1cd6610e061b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:05:46 +02:00
Thomas Wolf 3dea2b37ec Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  [test] Create keystore with the keytool of the running JDK

Change-Id: I981de862c614986a7b443fed1cce7b895b758682
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:04:26 +02:00
Thomas Wolf 4322dd26a3 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  [test] Create keystore with the keytool of the running JDK

Change-Id: Ic56f4f23c37432377be88e07d06c5ad75591d84f
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 13:03:08 +02:00
Thomas Wolf 6cb39d145a [test] Create keystore with the keytool of the running JDK
Call keytool with the absolute path of "java.home". Otherwise a keytool
for a different, maybe even newer Java version might be picked up, and
then the keystore may not be readable by the JVM used to run the tests.

(cherry picked from commit 2d73c702d3)

Change-Id: Iea77024947a34267f008847d81312fe0abadc615
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-29 12:47:16 +02:00
Thomas Wolf f1e67911ed [gpg] Update to Bouncy Castle 1.69
Bump lower bound in MANIFEST.MF and adapt code.

Change-Id: I3a3c7948e5fc29f5517fe84209fcea81834e8e5b
2021-07-26 13:30:37 +02:00
Thomas Wolf 2d73c702d3 [test] Create keystore with the keytool of the running JDK
Call keytool with the absolute path of "java.home". Otherwise a keytool
for a different, maybe even newer Java version might be picked up, and
then the keystore may not be readable by the JVM used to run the tests.

Change-Id: Iea77024947a34267f008847d81312fe0abadc615
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-25 15:48:26 +02:00
Thomas Wolf cbb9188cda [sshd] Minor code clean-up
Since upstream changed the method profile to throw Exception it's not
necessary anymore to re-throw as IOException.

Change-Id: I31afab4f6e1a2f0feef79e6abced20d0ca1c493b
2021-07-21 16:06:57 -04:00
Julian Ruppel e2798413f9 Support commit.template config property
Adds functionality to read the git commit.template property. The
template content is read either via a default encoding or, if present,
via encoding specified by i18n.commitEncoding property.

Bug: 446355

Change-Id: I0c45db98e324ddff26a7e0262835f259d6528a86
Signed-off-by: Julian Ruppel <julian.ruppel@sap.com>
2021-07-21 15:50:02 -04:00
Matthias Sohn ca969ecc61 Relax version range for hamcrest and assertj
We currently cannot use hamcrest 2.2 and assertj 3.20.2 (which requires
hamcrest 2.2) in egit tests since other Eclipse dependencies (e.g.
swtbot) require hamcrest 1.1.

Hence relax version range for these components in jgit so that jgit
tests also work when using the egit target platform.

Change-Id: I521e2ec4491bd8d790609b8a66a8f14511a865a1
2021-07-19 12:32:26 +02:00
Matthias Sohn 8089c8b39c bazel: don't expose jsch and jzlib to org.eclipse.jgit
These libraries are no longer used by org.eclipse.jgit since the code
depending on them was moved to org.eclipse.jgit.ssh.jsch.

Change-Id: Ic1c66e8ca06d904abf44d84154bbe9a770aa94d5
2021-07-16 23:47:30 +02:00
Matthias Sohn 02344254ea Update orbit to I20210713220109
update
- org.apache.commons.compress to 1.20.0.v20210713-192
- org.bouncycastle.bcpg to 1.69.0.v20210713-1924
- org.bouncycastle.bcpkix to 1.69.0.v20210713-1924
- org.bouncycastle.bcprov to 1.69.0.v20210713-1924

- add org.bouncycastle.bcutil 1.69.0.v20210713-1924

In bazel build don't expose bouncycastle to org.eclipse.jgit since it's
not used there anymore since code depending on bouncycastle was moved to
org.eclipse.jgit.gpg.bc.

CQ: 21771
CQ: 23471
CQ: 23472
CQ: 23473
CQ: 23474
Change-Id: Id3d94c00c39bbc57e3f49a61150841249dc3985c
2021-07-16 23:47:30 +02:00
Matthias Sohn e0ba98edd2 Update orbit to I20210711110031
and update
- assertj to 3.20.2.v20210706-1104
- hamcrest to 2.2.0.v20210711-0821
  - classes which were in org.hamcrest.core 1.3 and org.hamcrest.library
    1.3 were all moved to org.hamcrest in 2.2
  - the annotation org.hamcrest.Factory was removed and is no longer
    needed
  - junit 4.13 requires hamcrest-core and hamcrest-library 1.3 therefore
    keep them in the target platform

CQ: 23501
Change-Id: Ife871c0343b611be9203aed7f86577e85bbf5c95
2021-07-16 23:47:29 +02:00
Thomas Wolf 13777a3a62 [sshd] Ignore revoked keys in OpenSshServerKeyDatabase.lookup()
It makes no sense to return revoked keys.

Change-Id: I99eee1de3dba5c0c8d275b7c1a24053874b3cb03
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-16 08:45:24 +02:00
Thomas Wolf 4c5c3e9fb8 [sshd] Distinguish key type and signature algorithm for host key
Since the introduction of the rsa-sha2-512 and rsa-sha2-256 signature
types, the key type for RSA is no longer automatically the signature
algorithm. We re-order the list for the host key proposal such that
keys we already have are preferred; this minimizes warnings about new
host keys. When doing so, put all of rsa-sha2-512, rsa-sha2-256, and
ssh-rsa at the front, in that order, not just ssh-rsa.

This ensures that we do prefer RSA keys if we already have an RSA host
key, but at the same time we still prefer the stronger signature
algorithms over the weaker and deprecated SHA1-based ssh-rsa signature.
It also helps avoid a bug found in some Github versions where the Github
SSH server uses a rsa-sha2-512 signature even though ssh-rsa was
negotiated.[1]

[1] https://www.eclipse.org/forums/index.php/t/1108282/

Bug: 574635
Change-Id: I0a49dcfa0c2c93f23118c983cd0bc9e5a467d886
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-16 08:45:23 +02:00
Thomas Wolf 27a1fa1872 [sshd] Implement SSH config KexAlgorithms
Make the used KEX algorithms configurable via the ssh config. Also
implement adding algorithms not in the default set: since sshd 2.6.0
deprecated SHA1-based algorithms, it is possible that the default set
has not all available algorithms, so adding algorithms makes sense.

This enables users who have to use a git server that only supports
old SHA1-based key exchange methods to enable those methods in the
ssh config:

  KexAlgorithms +diffie-hellman-group1-sha1

There are two more SHA1 algorithms that are not enabled by default:
diffie-hellman-group14-sha1 and diffie-hellman-group-exchange-sha1.
KeyAlgorithms accepts a comma-separated list of algorithm names.

Since adding algorithms is now supported, adapt the handling of
signature algorithms, too. Make sure that definitions for the KEX
exchange signature (HostKeyAlgorithms) don't conflict with the
definition for signatures for pubkey auth (PubkeyAcceptedAlgorithms).

HostKeyAlgorithms updates the signature factories set on the session
to include the default factories plus any that might have been added
via the SSH config. Move the handling of PubkeyAcceptedAlgorithms
from the client to the JGitPubkeyAuthentication, where it can be done
only if pubkey auth is attempted at all and where it can store its
adapted list of factories locally.

Bug: 574636
Change-Id: Ia5d5f174bbc8e5b41e10ec2c25216d861174e7c3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-07-16 08:45:23 +02:00
Matthias Sohn 1e391d47ba Update orbit to S20210705204906
Change-Id: Ib4ec9f80a8a346fa8c709e58f8076372d14eca64
2021-07-08 21:54:38 +02:00
Matthias Sohn 729c92d723 Update jetty to 9.4.43.v20210629
Change-Id: I4ceea58cb0a04d041ca4e698e096a00ccd12bfc0
2021-07-08 18:15:04 +02:00
Matthias Sohn caf3fbe8bf Update JMH used in benchmarks to 1.32
Change-Id: I98734165de8256ab1461733169957efbfa5f36e4
CQ: 23499
CQ: 23500
2021-07-08 17:59:49 +02:00
Thomas Wolf 66c9c7bf87 Update target platform to I20210626190330
Update
- com.google.gson to 2.8.7.v20210624-1215
- javaewah to 1.1.12.v20210622-2206
- org.apache.sshd.osgi to 2.7.0.v20210623-0618
- org.apache.sshd.sftp to 2.7.0.v20210623-0618
- org.tukaani.xz to 1.9.0.v20210624-1259
- Apache MINA sshd to 2.7.0
  - Remove work-arounds for problems resolved upstream since 2.6.0, and
    adapt to upstream API changes.
- update DEPENDENCIES.

CQ: 23469
CQ: 23470
CQ: 23496
CQ: 23497
CQ: 23498
Bug: 574220
Change-Id: I898b216c3492f8488fbf25fa4b49f1250f86f3c8
Also-by: David Ostrovsky <david@ostrovsky.org>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2021-07-06 13:22:38 +02:00
Matthias Sohn 1c6bd377ff Add 4.21 target platform
Change-Id: I88d32956fd7ad5c23aa395abd88f35359b519b79
2021-07-01 20:15:21 +02:00
Matthias Sohn 4acedc5d2e Update 4.20 target platform to use final 4.20 release
Change-Id: I8d6a1617a7713faf5186cfb5b2a084d0f1f9e63f
2021-07-01 20:15:20 +02:00
Matthias Sohn 23ae9f827c Merge "Remove use of deprecated getAllRefs() in ReceivePack" 2021-07-01 14:13:19 -04:00
Matthias Sohn 06ca6cb3b0 Merge "Merge branch 'stable-5.11' into stable-5.12" into stable-5.12 2021-06-29 14:22:22 -04:00
Matthias Sohn 0b9193e69f [pgm] Fix default meta variable defined in StopOptionHandler
Change-Id: I703718e2476d40d34b11083414ac0ed5e34bf90a
2021-06-28 01:10:41 +02:00
Matthias Sohn d947e84980 Fix typo in javadoc of PathTreeFilterHandler
Change-Id: Icf0dc814baf0dc44addd102d9c03b821b4022a84
2021-06-28 01:08:48 +02:00
Luca Milanesio ed5be35e2e Remove use of deprecated getAllRefs() in ReceivePack
Repository.getAllRefs() is deprecated and should not
be used anymore. Leverage the ref-db and the retrieval
of refs by prefix and adapt the result to the expected
refname/ref map.

Bug: 534731
Change-Id: I37a9092859f220ddc4e5063d01544f3e82208be8
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
2021-06-26 16:41:01 +02:00
Matthias Sohn d46af8c69d Merge branch 'stable-5.12'
* stable-5.12:
  Retry loose object read upon "Stale file handle" exception
  Ignore missing javadoc in test bundles

Change-Id: I67c613c066a3252f9b0d0a3dcc026b57e10bfe1d
2021-06-26 16:37:59 +02:00
Matthias Sohn 8bd0161c83 Merge branch 'stable-5.11' into stable-5.12
* stable-5.11:
  Retry loose object read upon "Stale file handle" exception
  Ignore missing javadoc in test bundles

Change-Id: Ia4dc886c920cec3c9da86e1a90a0af68bd016b4f
2021-06-26 16:36:55 +02:00
Matthias Sohn e6ace4a96d Merge branch 'stable-5.10' into stable-5.11
* stable-5.10:
  Retry loose object read upon "Stale file handle" exception
  Ignore missing javadoc in test bundles

Change-Id: Ia385fa6b5d2fee64476793e06860a279bf2f6e36
2021-06-26 16:35:20 +02:00
Matthias Sohn bbb1c7f645 Merge branch 'stable-5.9' into stable-5.10
* stable-5.9:
  Retry loose object read upon "Stale file handle" exception
  Ignore missing javadoc in test bundles

Change-Id: I56fc2c47193a891285a705d44b3507f23982dc8a
2021-06-25 22:23:47 +02:00
Fabio Ponciroli 6976a30f44 searchForReuse might impact performance in large repositories
The search for reuse phase for *all* the objects scans *all*
the packfiles, looking for the best candidate to serve back to the
client.

This can lead to an expensive operation when the number of
packfiles and objects is high.

Add parameter "pack.searchForReuseTimeout" to limit the time spent
on this search.

Change-Id: I54f5cddb6796fdc93ad9585c2ab4b44854fa6c48
2021-06-25 17:57:59 +02:00
Antonio Barone 24d6d60538 Retry loose object read upon "Stale file handle" exception
When reading loose objects over NFS it is possible that the OS syscall
would fail with ESTALE errors: This happens when the open file
descriptor no longer refers to a valid file.

Notoriously it is possible to hit this scenario when git data is shared
among multiple clients, for example by multiple gerrit instances in HA.

If one of the two clients performs a GC operation that would cause the
packing and then the pruning of loose objects, the other client might
still hold a reference to those objects, which would cause an exception
to bubble up the stack.

The Linux NFS FAQ[1] (at point A.10), suggests that the proper way to
handle such ESTALE scenarios is to:

"[...] close the file or directory where the error occurred, and reopen
it so the NFS client can resolve the pathname again and retrieve the new
file handle."

In case of a stale file handle exception, we now attempt to read the
loose object again (up to 5 times), until we either succeed or encounter
a FileNotFoundException, in which case the search can continue to
Packfiles and alternates.

The limit of 5 provides an arbitrary upper bounds that is consistent to
the one chosen when handling stale file handles for packed-refs
files (see [2] for context).

[1] http://nfs.sourceforge.net/
[2] https://git.eclipse.org/r/c/jgit/jgit/+/54350

Bug: 573791
Change-Id: I9950002f772bbd8afeb9c6108391923be9d0ef51
2021-06-24 23:52:22 +02:00
Matthias Sohn 12f39c26b0 Ignore missing javadoc in test bundles
Change-Id: I83ed20823dc6b22ff48c2a554acb2f7d3b6067b7
2021-06-24 23:14:13 +02:00
andrewxian2000 64b0dee98f Fix garbage collection failing to delete pack file
The loosen() method has opened pack file and the open pack file handle
may prevent it from being deleted e.g. on Windows. Fix this by closing
the pack file only after loosen() finished.

Bug: 574178
Change-Id: Icd59931a218d84c9c97b450eea87b21ed01248ff
Signed-off-by: andrew.xian2000@gmail.com
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2021-06-24 11:03:27 +12:00
Marco Miller f598e69529 Upgrade maven-dependency-plugin to 3.2.0
Change-Id: I9dea176e8b30658bcb6a9dee8c54184c0fa86de8
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
2021-06-22 14:45:51 -04:00
Marco Miller 6692e39452 Upgrade jacoco-maven-plugin to 0.8.7
Change-Id: I6742b2d8ec75422542a55a9840bfe43e8cf17400
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
2021-06-22 14:45:36 -04:00
Marco Miller ffe55a8005 Upgrade maven-jxr-plugin to 3.1.1
Change-Id: I66d7589ea1b823305d908acb13367ef78ef55f7a
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
2021-06-22 14:45:08 -04:00
andrewxian2000 1ae7e08bca Fix garbage collection failing to delete pack file
The loosen() method has opened pack file and the open pack file handle
may prevent it from being deleted e.g. on Windows. Fix this by closing
the pack file only after loosen() finished.

Bug: 574178
Change-Id: Icd59931a218d84c9c97b450eea87b21ed01248ff
Signed-off-by: andrew.xian2000@gmail.com
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2021-06-22 14:16:30 +02:00
Thomas Wolf fc57689774 Fix PathSuffixFilter: can decide only on full paths
On a subtree, a PathSuffixFilter must return -1 ("indeterminate"),
not 0 ("include"), otherwise negation goes wrong: an indeterminate
result (-1) is passed on, but a decision (0/1) is inverted.

As a result a negated PathSuffixFilter would skip all folders.

Bug: 574253
Change-Id: I27fe785c0d772392a5b5efe0a7b1c9cafcb6e566
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2021-06-21 13:03:22 +02:00
Matthias Sohn 2fc600b7cc Update jetty to 9.4.42.v20210604
Change-Id: Ie0abae5686064ccfeb34e8af0d98471162036b00
2021-06-21 04:51:01 -04:00