Commit Graph

7271 Commits

Author SHA1 Message Date
Matthias Sohn 8bf9c668ad Wait opening new packfile until it can't be racy anymore
If
- pack.waitPreventRacyPack = true (default is false)
- packfile size > pack.minSizePreventRacyPack (default is 100 MB)
wait after a new packfile was written and before it is opened until it
cannot be racy anymore.

If a new packfile is accessed while it's still racy at least the pack's
index will be reread by ObjectDirectory.scanPacksImpl(). Hence it may
save resources to wait one tick of the file system timer to avoid this
reloading. On filesystems with a coarse timestamp resolution it may be
beneficial to skip this wait for small packfiles.

Bug: 546891
Change-Id: I0e8bf3d7677a025edd2e397dd2c9134ba59b1a18
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-04 18:15:17 +02:00
Matthias Sohn 692524d2bd Update to Orbit R20190602212107
This version fixed wrong
Require-Capability: osgi.ee; filter:="(&(osgi.ee=JavaSE)(version=9))"
in bouncycastle 1.61.

Bug: 547570
Change-Id: I715bcc4b39c3d4ad036dcc5bbdf48d321759f55f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-03 10:04:04 +02:00
David Pursehouse e8ed59a2ac PacketLineIn: Deprecate the END constant
Deprecate the constant with the intention of making it private in
a future release.

All existing usage of the constant within JGit code has already been
replaced with the recommended alternatives in preceding commits.

Change-Id: I10eb95f3f92cb74f93a26bf1a6edd24615b75c6f
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-03 08:07:24 +09:00
David Pursehouse ee747827b0 PacketLineIn: Add an iterator over strings in the input stream
Allows callers to read all lines in the input stream until the
END marker is reached, without having to explicitly check for
the END marker.

Replace all remaining usage of the END marker with the new method.

Change-Id: I51f419c7f569ab7ed01e1aaaf6b40ed8cdc2116b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-03 08:07:24 +09:00
David Pursehouse e0133b9440 Replace most usages of PacketLineIn.END with PacketLineIn.end()
PacketLineIn.END is only referenced in tests. Replace most of those
with a new package visible end() method.

Remaining usages of PacketLineIn.END are in the form:

  while ((line = pckIn.readString()) != PacketLineIn.END) {

and are not trivial replacements, hence are not touched in this change.

Change-Id: Id77c5321ddcad127130b246bde8f08736e60e1ea
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-03 08:07:24 +09:00
David Pursehouse e99aaa5f79 PacketLineIn: Deprecate DELIM constant
Deprecate DELIM with the intention of making it private in a future
release.

Callers that want to test if a packet line string is the delimiter
should use the isDelimiter(String) method.

The only other references to DELIM in the JGit code are in tests. For
those, introduce a package visible delimiter() method.

Change-Id: I21e8bbac0ffb9ef710c9753e23435416b09a4891
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-03 08:07:24 +09:00
David Pursehouse 7377b8dd01 Replace trivial reference comparison of PacketLineIn.{DELIM,END}
Replace reference comparisons of PacketLineIn's DELIM and END strings
with usage of the helper methods isDelimiter() and isEnd().

Change-Id: I52dcfc4ee9097f1bd6970601c716701847d9eebd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-02 18:32:59 +09:00
David Pursehouse 07a07a75d6 PacketLineIn: Rename isDelim to isDelimiter
Change-Id: I279808b8ddd8a9230a35582c00867b811633dfe8
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-02 18:32:55 +09:00
David Pursehouse 5862a5e222 ProtocolV2ParserTest: Fix typo in comment
Change-Id: I22b07179ba6e2517ae3e178c31fde20987cead34
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-02 11:18:55 +09:00
Brandon Weeks f4443ef3f9 Upgrade Bouncy Castle to 1.61
Also now use JcaKeyBoxBuilder constructor in
BouncyCastleGpgKeyLocator.readKeyBoxFile(Path).

CQ: 19868
CQ: 19869
CQ: 19870
Change-Id: I45bd80e158aecd90448b0c7e59615db27aaef892
Signed-off-by: Brandon Weeks <bweeks@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-01 01:39:26 +02:00
Matthias Sohn fc40a173d6 Update to Orbit R20190531194818 and rollback update to Ant 1.10.6
An issue has been reported with Java 1.8 and new Ant version 1.10.6
hence this was rolled back to 1.10.5 in Orbit.

See: https://bz.apache.org/bugzilla/show_bug.cgi?id=63457
Bug: 547607
Change-Id: I9c04c988ec89bb1e2cbb528a2d3add30a0b7b3b9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-01 00:58:31 +02:00
Sebastian Schuberth 9ea1ed158b cli: Add the --always option to describe
Change-Id: I0342d589e4deabe9d80ea3c9c6b48d7b265d8fe6
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2019-05-31 17:11:17 +02:00
Sebastian Schuberth 237c76fddf DescribeCommand: Support the "always" option
See:

https://git-scm.com/docs/git-describe#Documentation/git-describe.txt---always

Extend the tests accordingly.

Change-Id: Ibfcda338a246c8cba0df6b6e7b9bad76c9f8b593
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2019-05-31 17:11:15 +02:00
Sebastian Schuberth 0cac834668 cli: Add the --tags option to describe
Change-Id: I78924e61e2050eeaff991ee56715f36514100492
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2019-05-31 17:10:13 +02:00
Sebastian Schuberth afcb3a8c9e DescribeCommand: Consistenly omit the default value
Omit the default value of "false" for "useTags" like already done for
"longDesc".

Change-Id: I25aaacae028fc8cf27f4040ba45fe79609318aa1
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2019-05-31 17:08:39 +02:00
Matthias Sohn 22352e4751 Remove excess blank line in FileUtilsTest
Change-Id: I0d617feb7baacf9917b1da605fcb75f7d4ccd184
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-31 09:40:59 +09:00
David Pursehouse d2425e16f4 PacketLineIn: Add helper methods to check for END and DELIM
These methods will allow clients to check for END and DELIM without
doing a reference comparison on the String objects, which raises
warnings from Error Prone.

Change-Id: I9e7e59843553ed4488ee8e864033198bbb60d67c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-30 20:31:39 +09:00
David Pursehouse 9f5d271b12 UploadPackTest: Rename variable to avoid hiding class member
Change-Id: I9e02642b2f56a735681e58b8c8d6e3545ef9093b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-30 20:18:28 +09:00
David Pursehouse cd1d2d6fe3 UploadPackTest: Add missing <> operator on instantiation of ArrayList
Change-Id: I348cbcd75252d7857ada843dc496d445ee1d62fb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-30 20:17:24 +09:00
Matthias Sohn 4904a625c9 Avoid null PackConfig in GC
Initialize it using the repository's config already in the constructor.

Change-Id: I4ea620a7db72956e7109f739990f09644640206b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-29 23:41:46 +02:00
Matthias Sohn 5d0286eb7c Add FileSnapshot test testing recognition of file size changes
Change-Id: Ibcd76a5e6e4183ada0be1d4436ce957243f2094d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-29 23:41:45 +02:00
Matthias Sohn 43b06f51f9 Capture reason for result of FileSnapshot#isModified
This allows to verify the expected behavior in
FileSnapshotTest#testSimulatePackfileReplacement and enables extending
FileSnapshot for packfiles to read the packfile's checksum as another
criterion to detect modifications without reading the full content.

Also add another field capturing the result of the last check if
lastModified was racily clean.

Remove unnecessary determination of raciness in the constructor. It was
determined twice in all relevant cases.

Change-Id: I100a2f49d7949693d7b72daa89437e166f1dc107
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-29 23:41:45 +02:00
Matthias Sohn d31a0c0b5d Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
NTFS does not support FileKey hence ignore this test on Windows.

Change-Id: I7b53a591daa5e03eb5e401b5b26d612ab68ce10d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-29 23:41:43 +02:00
Matthias Sohn 1e8b68cbc5 Tune max heap size for tests
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].

Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test

Decrease it to 768m for
- org.eclipse.jgit.test

[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console

Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit e19e859977)
2019-05-29 23:39:55 +02:00
Matthias Sohn 0e33b32ab7 Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
testNewFileNoWait() was identical to testNewFileWithWait() but claims it
doesn't wait at all. Hence remove the waits.

Change-Id: I49b8ca5cb49a43c55fe61870c18c42f32fb4b74d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-27 10:49:51 +02:00
Marc Strapetz f1577909e7 ObjectDirectory: fix closing of obsolete packs
This resolves a regression introduced in fef78212.

Change-Id: Ibb4521635a87012520566efc70870c59d11be874
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2019-05-27 10:49:51 +02:00
Ivan Frade 82316fc23a BitmapCalculator: javadoc fixes
Fill out the description of when IOException is thrown.

Also fix a typo in the description for IncorrectObjectTypeException.

Change-Id: I9fafd19d68ddc4fe4e95e8516c2c38484b941a3a
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-27 10:38:37 +09:00
Matthias Sohn 92d1ca8099 Update API filters for methods added to fix bugs
Change-Id: I235c608d80a14c8af39400ae2e749488a4fa0d22
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-26 20:54:44 +02:00
David Pursehouse 245c2c09a0 Bazel: Increase severity of most error-prone checks to ERROR
Most of the checks can be increased to ERROR because there is no
code in the project that triggers them.

There are still several that are triggered, and these are left
at the WARN severity:

  https://errorprone.info/bugpattern/AmbiguousMethodReference
  https://errorprone.info/bugpattern/BadComparable
  https://errorprone.info/bugpattern/ClassNewInstance
  https://errorprone.info/bugpattern/Finally
  https://errorprone.info/bugpattern/FutureReturnValueIgnored
  https://errorprone.info/bugpattern/ImmutableEnumChecker
  https://errorprone.info/bugpattern/NarrowingCompoundAssignment
  https://errorprone.info/bugpattern/NonOverridingEquals
  https://errorprone.info/bugpattern/OperatorPrecedence
  https://errorprone.info/bugpattern/ReferenceEquality
  https://errorprone.info/bugpattern/ShortCircuitBoolean
  https://errorprone.info/bugpattern/StringEquality
  https://errorprone.info/bugpattern/TypeParameterUnusedInFormals

These can be cleaned up and increased to ERROR in follow-up
commits.

Change-Id: Icfc3b3163e129e504f10b3fc856aef262f723f99
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-26 18:03:07 +09:00
David Pursehouse 4c33f70d5a Enable error-prone checks by default
Based on recent work by Dave Borowitz on the Gerrit project [1, 2].

The warnings/error configuration is unchanged, but now the checks are
enabled by default during the build rather than having to be manually
invoked.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/225653
[2] https://gerrit-review.googlesource.com/c/gerrit/+/225714

Change-Id: I7f4849896af72de38880b13d64519cc93bb3d2a3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-26 17:47:08 +09:00
David Ostrovsky a4f5fc1572 Add bazel options to align with gerrit's
Change-Id: I6addd5fcaecb8c2508e1d5776ecfed8ae676b6ed
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2019-05-26 02:44:33 -04:00
Andrey Loskutov e57a37e767 RevWalkUtils: add progress callback to findBranchesReachableFrom
Offer a version of findBranchesReachableFrom method with progress
monitor callback. This is required to allow UI clients to cancel long
running operations and show progress.

Bug: 547642
Change-Id: I31d1de54dbaa6ffb11e03da4c447963e8defa1d0
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2019-05-25 13:52:49 +02:00
David Pursehouse 76c867d62a Merge branch 'stable-5.4'
* stable-5.4:
  SystemReader: Respect passed-in parent when no system config is present
  Prepare 5.4.0-SNAPSHOT builds
  JGit v5.4.0.201905221418-m3
  Prepare 5.4.0-SNAPSHOT builds
  JGit v5.4.0.201905081430-m2

Change-Id: If69cb0afe9915b926d977e4ad26c7d72db31667c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-24 08:34:40 +09:00
David Pursehouse 7eb222f20d Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  SystemReader: Respect passed-in parent when no system config is present

Change-Id: I758ed0c2a3527ffcf6255158a49090aba090e942
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-24 08:33:45 +09:00
Dave Borowitz 04bd5821e9 SystemReader: Respect passed-in parent when no system config is present
The default implementation of openSystemConfig has special handling for
when the FS returns null from getGitSystemConfig: it should return a
"real" FileBasedConfig instance that never actually tries to load a
file. However, this codepath was not respecting the passed-in parent
config.

Change-Id: Id0bcdc93bb42f9ebe3f5ee4c6b4be8863e0133f9
2019-05-23 16:07:49 -07:00
David Pursehouse 87c7bda914 Upgrade maven-source-plugin to 3.1.0
Change-Id: I8253e75f5038b6c0e2205d74683e0ebd9839dd33
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-23 10:13:13 +09:00
David Pursehouse d510929a25 Upgrade maven-jar-plugin to 3.1.2
Change-Id: I9a291390205ee8c44b02c4081f2d8f17635d90ec
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-23 10:13:13 +09:00
David Pursehouse 22360a4bee Upgrade jacoco-maven-plugin to 0.8.4
Change-Id: Ibab24e1f5eefdc14bcf9ab6d72eab7e0a0cfaad9
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-23 10:13:13 +09:00
Matthias Sohn 73ba6658cf BitmapCalculator and its test: add missing license header
Change-Id: I3c4a8b1e9159b0553aa95213bb82628370b6c036
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-23 02:12:16 +02:00
Matthias Sohn 0988276746 Prepare 5.4.0-SNAPSHOT builds
Change-Id: I9df6fccee253d4087f9afab4cb46e0a40b8a5699
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 22:52:43 +02:00
Matthias Sohn 72998dd5ee JGit v5.4.0.201905221418-m3
Change-Id: I8a7e687acfcf8c341abd726ae60b5ee173422215
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 20:18:14 +02:00
Ivan Frade 5dce8614ab RevWalk: new method createReachabilityChecker()
Every caller would need to check if bitmaps are available in the repo to
instantiate a reachability checker.

Offer a method to create the reachability checker in the walk: the
caller has already a walk over the repo, and the walk has all the
information required.

This allows us to make the implementation classes package-private.

Change-Id: I355e47486fcd9d55baa7cb5700ec08dcc230eea5
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-05-22 14:17:31 -04:00
Matthias Sohn bed7a45fb5 Merge branch 'master' into stable-5.4
* master:
  ReachabilityCheckers: @since 5.4
  Update Orbit to S20190521195709 for 2019-06 M3
  Tune max heap size for tests
  GPG: check that the key found is a signing key
  GPG: use key fingerprint suffix to compare id for signing key
  Recognize ReachabilityCheckerTestCase as helper
  UploadPack: restore inadvertently deleted line
  UploadPack: Use reachability checker to validate non-advertised wants
  Bazel: Simplify names of bouncy castle libraries
  Convert to lambda or member reference
  BitmappedReachabilityChecker: Reachability check using bitmaps
  BitmapCalculator: Get the reachability bitmap of a commit
  ReachabilityChecker: Default implementation with a RevWalk

Change-Id: I323c460853230abcffa94c4f34f15dbaff8c330c
2019-05-22 19:37:40 +02:00
Ivan Frade 8216929618 ReachabilityCheckers: @since 5.4
The new ReachabilityChecker interface and its implementations are marked
as @since 5.5, but they will make it to the 5.4 release.

Change-Id: I88c31b3300ccf35d18c35faddb2517f0a57bdcfd
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-05-22 17:37:12 +02:00
Matthias Sohn c33c0bd1b5 Update Orbit to S20190521195709 for 2019-06 M3
Update
- org.apache.ant to 1.10.6.v20190516-0412 (CQ 19829)
- org.apache.httpcomponents.httpclient to 4.5.6.v20190503-0009

CQ: 19829
Change-Id: Ia057d4f07dab21374ca66c3d87a2bab1c795562c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 17:34:58 +02:00
Matthias Sohn e19e859977 Tune max heap size for tests
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].

Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test

Decrease it to 768m for
- org.eclipse.jgit.test

[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console

Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 14:42:43 +02:00
Matthias Sohn 91101414ae Include filekey file attribute when comparing FileSnapshots
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).

Some filesystems expose unique file identifiers, e.g. inodes in Posix
filesystems which are named filekeys in Java's BasicFileAttributes. Use
them as another means to detect file modifications based on stat
information.

Running git gc on a repository yields a new packfile with the same id as
a packfile which existed before the gc if these packfiles contain the
same set of objects. The content of the old and the new packfile might
differ if a different PackConfig was used when writing the packfile.
Considering filekeys in FileSnapshot may help to detect such packfile
modifications.

Bug: 546891
Change-Id: I711a80328c55e1a31171d540880b8e80ec1fe095
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 08:13:56 +02:00
Matthias Sohn b513b77477 Measure file timestamp resolution used in FileSnapshot
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].

Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.

Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.

Measured Java file timestamp resolution: [2]

[1] https://github.com/git/git/blob/master/Documentation/technical/racy-git.txt
[2] https://docs.google.com/spreadsheets/d/1imy0y6WmRqBf0kjCxzxj2X7M50eIVfa7oaUIzEOHmjo

Bug: 546891
Change-Id: I493f3b57b6b306285ffa7d392339d253e5966ab8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 08:13:22 +02:00
Matthias Sohn 201bbd6ead Fix FileSnapshot's consideration of file size
* fix equals() and hashCode() methods to consider size
* fix toString() to show size

Change-Id: I5485db55eda5110121efd65d86c7166b3b2e93d0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 08:12:51 +02:00
Matthias Sohn ed2e515ddf Fix API problem filters
Change-Id: I566391d7c51875f30cf580d64e6784819985709f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 08:03:29 +02:00