Commit Graph

7271 Commits

Author SHA1 Message Date
Matthias Sohn 2d84bb4341 Improve retry handling when saving FileStoreAttributes fails
- fix handling of interrupts in FileStoreAttributes#saveToConfig
- increase retry wait time to 100ms
- don't wait after last retry
- dont retry if failure is caused by another exception than
LockFailedException

Change-Id: I108c012717d2bcce71f2c6cb9cf0879de704ebc2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-13 08:33:32 +02:00
David Pursehouse cb53db4bff Merge "Ensure FSTest uses MockSystemReader" into stable-5.1 2019-08-12 19:37:11 -04:00
Matthias Sohn 69ce7fc7fb Ensure FSTest uses MockSystemReader
Tests shall not modify ~/.gitconfig. When running tests with bazel this
test failed since bazel isolates tests in a sandbox.

Change-Id: I7dd092afd14972da58a95eb7c200d353f0959fa1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-13 00:26:41 +02:00
Vishal Devgire 9b4a55c0b6 Make supportsAtomicCreateNewFile return true as default
The method org.eclipse.jgit.util.FS.supportsAtomicCreateNewFile()
should default to true as mentioned in docs [1]

org.eclipse.jgit.util.FS_POSIX.supportsAtomicCreateNewFile() method
will set the value to false if the git config
core.supportsatomiccreatenewfile is not set.

It should default to true if the configuration is undefined.

[1]
4169a95a65/org.eclipse.jgit/src/org/eclipse/jgit/util/FS_POSIX.java (L372)

Bug: 544164
Change-Id: I16ccf989a89da2cf4975c200b3228b25ba4c0d55
Signed-off-by: Vishal Devgire <vishaldevgire@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-12 14:35:06 +02:00
Matthias Sohn 4d46a68abe Update orbit to R20190602212107-2019-06 to enable backports from master
update
- org.apache.httpcomponents.httpclient to 4.5.6.v20190503-0009
- org.apache.httpcomponents.httpcore to 4.4.10.v20190123-2214
- com.jcraft.jsch" version to 0.1.55.v20190404-1902
- org.mockito to 2.23.0.v20190527-1420
  add its dependencies
    - net.bytebuddy.byte-buddy 1.9.0.v20181107-1410
    - net.bytebuddy.byte-buddy-agent 1.9.0.v20181106-1534
- org.objenesis to 2.6.0.v20180420-1519

CQ: 18303 (objenesis)
CQ: 18703 (httpclient)
CQ: 18704 (httpcore)
CQ: 19588 (jsch)
CQ: 20214 (mockito)
CQ: 20215 (butebuddy)
CQ: 20216 (bytebuddy-agent)
Change-Id: I8d76375de867e32b7818449f843b7c053a717b5e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-12 14:35:06 +02:00
Matthias Sohn 6ef225c519 Handle InvalidPathException in FS_POSIX#createNewFileAtomic
Bug: 547400
Change-Id: Ic3541e360a2968ba3532a3d3fa4828b0d0463c02
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-12 12:18:16 +02:00
Matthias Sohn cc29da8e0f Ensure root cause of lock creation failures is logged
Change-Id: I91cdf1e085a29c0aabd6d22c6ebe848b2d75f42c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-12 12:17:11 +02:00
Matthias Sohn 9add6d3e24 Implement toString in MockSystemReader and MockConfig
This helps when debugging.

Change-Id: I3d72e1ea207ba60be77a7a9a840abb71ade1271c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-11 00:06:03 +02:00
Matthias Sohn 118ee61640 LocalDiskRefTreeDatabaseTest shall use MockSystemReader
It missed to call the setup() method of its super class which prepares
the MockSystemReader

Change-Id: I39858749f8d0115fc6ac7edc8847ffb2bbc85c33
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-11 00:06:03 +02:00
Matthias Sohn c28167e103 Ensure LocalDiskRepositoryTestCase#setup fully uses MockSystemReader
FS#getFileStoreAttributes used the real userConfig and not the mocked
one. This led to test errors when running tests with Bazel since it
sandboxes tests which prevents they can write to ~/.gitconfig.

Fix this by first preparing the MockedSystemReader and the mocked config
before calling FS#getFileStoreAttributes.

Also fix ConfigTest which broke due to this change since it inherits
from LocalDiskRepositoryTestCase and calls its setup method which was
changed here. We can no longer assert by comparing plain text since FS
adds FileStoreAttributes to the mocked userConfig. Also the default
options seen by this test changed since we now use a mocked config.

Change-Id: I76bc7c94953fe979266147d3b309a68dda9d4dfe
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-11 00:06:03 +02:00
Matthias Sohn 12d03a97d8 Ensure we use MockSystemReader in tests
If we use the default system reader FileStoreAttributes cannot persist
attributes in userConfig when tests run in Bazel due to sandboxing.
Hence we need to ensure that all tests use MockSystemReader (and
especially a mocked userConfig).

Change-Id: Ic1ad8e2ec5a150c5433434a5f6667d6c4674c87d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 23:57:21 +02:00
Matthias Sohn 65cee6ccbf Override FileBasedConfig's save method in MockConfig
This ensures we don't try to persist MockConfig using its superclasses
save() method which fails with an NPE since MockConfig has no backing
file.

Change-Id: Ifba2d24c9438bb30d3828ed31a4c131f940b45eb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 23:57:21 +02:00
Matthias Sohn a7338dd1e5 Remove FileBasedConfig.load(boolean) introduced in d45219ba
We can't add this method to the super class StoredConfig since that
abstracts from filesystem storage. MockSystemReader.MockConfig is a
StoredConfig and is also used by tests for dfs based storage. Hence
remove this leaky abstraction.

This implies we always use the fallback FileStoreAttributes which means
a config file modification is considered racy within the first 2
seconds. This should not be an issue since typically configs change
rarely and re-reading a config within the racy period is relatively
cheap since configs are small.

Change-Id: Ia2615addc24a7cadf3c566ee842c6f4f07e159a5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 23:57:21 +02:00
Matthias Sohn 0bc89fabe2 Disable debug log for FS in org.eclipse.jgit.test
This was enabled unintentionally in 06fc6c7c and spams the test logs. We
can enable this when needed.

Change-Id: I9f3042c0e285ff236be65fcc02bdcfdb90efc3af
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 20:32:07 +02:00
Matthias Sohn 06fc6c7cc0 Bazel: enable logging for tests in org.eclipse.jgit.test
- use slf4j-simple for logging in test runs
- for log configuration see
https://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html

Change-Id: I9f0a532644b31162c867cd0d63f083296eaf6be5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 01:21:18 +02:00
Matthias Sohn 42000a71e3 LockFile: log exception if creation of lock file failed
Change-Id: I4de75d12ec9e61193494916307289378cdb6220e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 00:57:04 +02:00
Thomas Wolf db0eb9f8ae GPG: check secring.gpg for keys if pubring.kbx is empty
If no OpenPGP key is found in pubring.kbx, try the legacy secring.gpg.
This appears to be consistent with GPG[1].

[1] https://lists.gnupg.org/pipermail/gnupg-users/2015-December/054881.html

Bug: 549439
Change-Id: I1557fd9b1f555a9b521fcd57cd3caccbdbacbeda
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-08-09 16:53:31 -04:00
Matthias Sohn 3cea3676c7 Merge branch 'stable-5.4'
* stable-5.4: (82 commits)
  Export all packages of o.e.j.ant and o.e.j.archive bundles
  Do not require test bundles to export all packages
  Fix API problem filters
  Increase severity of AmbiguousMethodReference to ERROR
  [error prone] suppress AmbiguousMethodReference in AnyLongObjectId
  [error prone] fix ReferenceEquality warning in CommitBuilder
  [error prone] suppress NonAtomicVolatileUpdate warning in SimpleLruCache
  [error prone] fix ReferenceEquality warning in CommitGraphPane#authorFor
  [error prone] fix ReferenceEquality warning in RevWalk#isMergedInto
  [error prone] fix ReferenceEquality warning in RefUpdate#updateImpl
  [error prone] fix ReferenceEquality warning in static equals methods
  [error prone] suppress AmbiguousMethodReference in AnyObjectId
  [error prone] fix "FutureReturnValueIgnored" error in FS
  Fix formatting and add missing braces in Repository#stripWorkDir
  Repository: fix reference comparison of Files
  MergeAlgorithm: Suppress Error Prone warning about reference equality
  Fix NarrowingCompoundAssignment warnings from Error Prone
  FS_POSIX: handle Files.getFileStore() failures
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  GlobalBundleCache: Fix ClassNewInstance warning from Error Prone
  IncorrectObjectTypeException: Fix typos in constructors' Javadoc
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Error Prone: Increase severity of NonOverridingEquals to ERROR
  Error Prone: Increase severity of ImmutableEnumChecker to ERROR
  GitDateParser#ParseableSimpleDateFormat: Make formatStr private final
  BatchRefUpdateTest: Suppress ImmutableEnumChecker warning
  PacketLineIn: Suppress comparison warnings for END and DELIM
  FileSnapshot#toString: Suppress ReferenceEquality warnings
  Blame: Suppress ReferenceEquality warning for RevCommit instances
  Fix API problem filters
  pgm: add missing optional dependency to org.tukaani:xz
  NetscapeCookieFile: Make hash static and group overloaded write
  NetscapeCookieFile: Javadoc fixes
  Config: Handle reference-equality warning (and empty javadoc)
  Error Prone: Increase severity of ShortCircuitBoolean to ERROR
  ObjectWalk: Prefer boolean operators over logical operators in comparisons
  BasePackFetchConnection: Prefer boolean operators over logical operators in comparisons
  PackWriter: Prefer boolean operators over logical operators in comparisons

Change-Id: I825fd55bcb5345fb7afe066bf54ca50325f40acb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 14:32:02 +02:00
Matthias Sohn a65e7c7501 Export all packages of o.e.j.ant and o.e.j.archive bundles
Follow best practice in Eclipse OSGi based projects to export all
packages to foster reuse and experimentation.

Change-Id: I27f2810fbf0439fcb7c907e7b4d570a9613f8aa6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 13:24:50 +02:00
Matthias Sohn 1574353f7c Do not require test bundles to export all packages
Configure PDE to not warn about not exported packages for test bundles.

Reusable test code which other bundles may want to reuse should go into
one of the junit bundles:
- org.eclipse.jgit.junit
- org.eclipse.jgit.junit.http
- org.eclipse.jgit.junit.ssh

Change-Id: Ifbdf30f8552e8f4bad6bbdeb63f729c53c844315
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 13:17:00 +02:00
Matthias Sohn 83502bb201 Fix API problem filters
- add missing filters for methods introduced in 5.4.1 to fix error prone
warnings
- remove no longer needed filters

Change-Id: Ia74d4e9876eae8acd403c8dea1fdf1227d68037e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 13:12:59 +02:00
David Pursehouse 40f6c92c5c Increase severity of AmbiguousMethodReference to ERROR
All instances of this problem have been fixed. Increase its severity
to ERROR to prevent reoccurrences.

Change-Id: I42d41a7c32b43d1ba59a28cd2f5a7d0ad315d8d9
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-09 16:09:04 +09:00
David Pursehouse 13df53c149 [error prone] suppress AmbiguousMethodReference in AnyLongObjectId
Move the implementation of the static equals() method to a new method
and suppress the error. Deprecate the old method to signal that we
intend to remove it in the next major release.

See https://errorprone.info/bugpattern/AmbiguousMethodReference

Change-Id: I712697a411ab44c6e05ae4604eb2dcb9c0f8abd3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-09 16:09:04 +09:00
David Pursehouse ad02a7cbad Stop using deprecated Constants.CHARACTER_ENCODING
Change-Id: I105b8a05bc64f249879a0795a059958553cc60c6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-09 12:11:41 +09:00
David Pursehouse 86be1b4038 Upgrade eclipse-jarsigner-plugin to 1.1.6
Change-Id: Iae7ebfe24bec19c0b56a325ab38384cdb05d3d90
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-09 10:01:39 +09:00
David Pursehouse 8ad6a7c61c Upgrade Spotbugs to 3.1.12.2
Change-Id: I4d673e058317e90b3de63f68202e580c0573f925
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-09 10:01:12 +09:00
Marco Miller 8974e147b7 Upgrade maven-site-plugin to 3.8.2
Change-Id: I79a3032e2ff7af14a41e6eb2618d2b1e22d1fde2
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-09 10:01:02 +09:00
Matthias Sohn 4b9ed3f678 [error prone] fix ReferenceEquality warning in CommitBuilder
Comparing with UTF_8 constant in StandardCharsets doesn't require to use
equals.

Change-Id: I6c73a929367f32c9e76ce99f6c0af268480d9230
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 01:50:01 +02:00
Matthias Sohn b4cb06b294 [error prone] suppress NonAtomicVolatileUpdate warning in SimpleLruCache
It's not important to update time field, scalability is more important
than perfect LRU ordering of cache entries.

Change-Id: I22466c580cd3613b81e1989130b2724af9d6c466
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 01:49:59 +02:00
Matthias Sohn 66cb2d9db4 [error prone] fix ReferenceEquality warning in CommitGraphPane#authorFor
Change-Id: I4d620ca65f7cd85863fe8b7182b01d262fbe3504
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 01:49:59 +02:00
Matthias Sohn 70258a9cb2 [error prone] fix ReferenceEquality warning in RevWalk#isMergedInto
Change-Id: Ibef75e2bc76e90f6e29c4cb3ba1c1f6e67009b10
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 01:49:59 +02:00
Matthias Sohn 722deaa671 [error prone] fix ReferenceEquality warning in RefUpdate#updateImpl
Change-Id: I6687e1eec6b6ecfe319a598ef8aec1976cee354b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 01:49:59 +02:00
Matthias Sohn bac0e8fd8d [error prone] fix ReferenceEquality warning in static equals methods
Implement a helper method suppressing the ReferenceEquality error prone
warning and use it to fix this warning in static equals methods where
this comparison is used to implement fast path of static equals
implementation.

See https://errorprone.info/bugpattern/ReferenceEquality

Change-Id: I33538a3406007d24efec3a504e031ca1069572ed
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-09 01:49:56 +02:00
Matthias Sohn 8f7e851346 [error prone] suppress AmbiguousMethodReference in AnyObjectId
Move the implementation of the static equals() method to a new method
and suppress the error. Deprecate the old method to signal that we
intend to remove it in the next major release.

See https://errorprone.info/bugpattern/AmbiguousMethodReference

Change-Id: I5e29c97f4db3e11770be589a6ccd785e2c9ac7f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 23:39:23 +02:00
Matthias Sohn 2f8911181b [error prone] fix "FutureReturnValueIgnored" error in FS
Change-Id: I53731091b3e34ac2e93a18f0ad6dd04dc56f8177
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 14:42:25 +02:00
Matthias Sohn a52b331d71 Fix formatting and add missing braces in Repository#stripWorkDir
Change-Id: I601d917f3741e0207a8ee7a365d9c2dea6422401
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 14:42:14 +02:00
Matthias Sohn 400bfd6665 Repository: fix reference comparison of Files
Change-Id: Ib46ea2c0d5039c88b4fc59723135c503a8c950ce
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 14:41:16 +02:00
David Pursehouse 6370098e54 MergeAlgorithm: Suppress Error Prone warning about reference equality
Error Prone reports:

  [ReferenceEquality] Comparison using reference equality instead of
  value equality

The END_EDIT instance is used as a marker, and thus it's OK to use
a reference equality comparison. Factor the comparison to a method
and add a suppression.

Change-Id: I7d9dc1fa21f46c984787056b0b5d163e313026a6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-08 14:34:24 +02:00
David Pursehouse 72ae089206 Fix NarrowingCompoundAssignment warnings from Error Prone
Error Prone reports:

  [NarrowingCompoundAssignment] Compound assignments from long to int
  hide lossy casts

and

  [NarrowingCompoundAssignment] Compound assignments from int to byte
  hide lossy casts

  See https://errorprone.info/bugpattern/NarrowingCompoundAssignment

Fix the warnings by adding explicit casts or changing types as
necessary.

Now that all occurrences of the warning are fixed, increase its
severity to ERROR.

Change-Id: Idb3670e6047b146ae37daee07212ff9455512623
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-08 14:28:27 +02:00
Thomas Wolf aefb11298c FS_POSIX: handle Files.getFileStore() failures
Android unconditionally throws a SecurityException;[1] getFileStore()
is not supported. Catch the exception and don't attempt the hard-
linking atomic file mechanism.

[1] https://android.googlesource.com/platform/libcore/+/21e6175e25

Bug: 548947
Change-Id: Idfba2d9dbcbc80ea15ab2ae7889e5142444c1581
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-08-08 14:14:38 +02:00
Matthias Sohn f54db4a857 Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: Iec3ad6ccc194582cb844310dc172c3103dae4457
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 13:24:14 +02:00
Matthias Sohn 718555518d Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of
StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: I238adfd3080a5fed9d64c3c757297da6ea893918
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 13:04:38 +02:00
Matthias Sohn 0046b2a8fe Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: Ia57385b2a60f48a5317c8d723721c235d7043a84
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 11:54:56 +02:00
Matthias Sohn 5a88815b1c Fix OpenSshConfigTest#config
- use FS.DETECTED instead of db.getFS() since the ssh config is
typically in a different place than the repository, the same is used in
OpenSshConfig
- reduce unnecessary repeated writes by introducing wait for one tick of
the file time resolution

Change-Id: Ifac915e97ff420ec5cf8e2f162e351f9f51b6b14
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 10:10:12 +02:00
David Pursehouse 1b22f1a1d6 Merge changes from topic 'sshd'
* changes:
  sshd: add missing javadoc in SshTestGitServer
  sshd: shared reference in JGitClientSession must be volatile
  sshd: correct the protocol version exchange
2019-08-07 21:31:57 -04:00
Han-Wen Nienhuys e60b9e1879 FileSnapshot: fix bug with timestamp thresholding
Increase the safety factor to 2.5x for extra safety if max of measured
timestamp resolution and measured minimal racy threshold is < 100ms, use
1.25 otherwise since for large filesystem resolution values the
influence of finite resolution of the system clock should be negligible.

Before, not yet using the newly introduced minRacyThreshold measurement,
the threshold was 1.1x FS resolution, and we could issue the
following sequence of events,

  start
  create-file
  read-file (currentTime)
  end

which had the following timestamps:

  create-file 1564589081998
  start 1564589082002
  read 1564589082003
  end 1564589082004

In this case, the difference between create-file and read is 5ms,
which exceeded the 4ms FS resolution, even though the events together
took just 2ms of runtime.

Reproduce with:
  bazel test --runs_per_test=100 \
    //org.eclipse.jgit.test:org_eclipse_jgit_internal_storage_file_FileSnapshotTest

The file system timestamp resolution is 4ms in this case.

This code assumes that the kernel and the JVM use the same clock that
is synchronized with the file system clock. This seems plausible,
given the resolution of System.currentTimeMillis() and the latency for
a gettimeofday system call (typically ~1us), but it would be good to
justify this with specifications.

Also cover a source of flakiness: if the test runs under extreme load,
then we could have

  start
  create-file
  <long delay>
  read
  end

which would register as an unmodified file. Avoid this by skipping the
test if end-start is too big.

[msohn]:
- downported from master to stable-5.1
- skip test if resolution is below 10ms
- adjust safety factor to 1.25 for resolutions above 100ms

Change-Id: I87d2cf035e01c44b7ba8364c410a860aa8e312ef
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-07 13:56:08 +02:00
Matthias Sohn 3b368d5578 In LockFile#waitForStatChange wait in units of file time resolution
Since we now measure file time resolution we can use it to replace the
hard coded wait time of 25ms. FileSnapshot#equals will return true until
the mtime of the old (o) and the new FileSnapshot (n) differ by at least
one  file time resolution.

Change-Id: Icb713a80ce9eb929242ed083406bfb6650c72223
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-07 13:21:39 +02:00
Matthias Sohn 6857138e19 Cache FileStoreAttributeCache per directory
Cache FileStoreAttributeCache entries since looking up FileStore for a
file may be expensive on some platforms.

Implement a simple LRU cache based on ConcurrentHashMap using a simple
long counter to order access to cache entries.

Change-Id: I4881fa938ad2f17712c05da857838073a2fc4ddb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
2019-08-06 14:54:39 +02:00
Matthias Sohn 275f3da783 Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
Use the fallback timestamp resolution as already described in the
javadoc of these methods. Using zero file timestamp resolution doesn't
make sense.

Change-Id: Iaad2a0f99c3be3678e94980a0a368181b6aed38c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-06 14:54:39 +02:00
Matthias Sohn d45219baac Persist minimal racy threshold and allow manual configuration
To enable persisting the minimal racy threshold per FileStore add a
new config option to the user global git configuration:

- Config section is "filesystem"
- Config subsection is concatenation of
  - Java vendor (system property "java.vendor")
  - Java version (system property "java.version")
  - FileStore's name, on Windows we use the attribute volume:vsn instead
    since  the name is not necessarily unique.
  - separated by '|'
  e.g.
    "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
  The same prefix is used as for filesystem timestamp resolution, so
  both values are stored in the same config section
- The config key for minmal racy threshold is "minRacyThreshold" as a
  time value, supported time units are those supported by
  DefaultTypedConfigGetter#getTimeUnit
- measure for 3 seconds to limit runtime which depends on hardware, OS
  and Java version being used

If the minimal racy threshold is configured for a given FileStore the
configured value is used instead of measuring it.

When the minimal racy threshold was measured it is persisted in the user
global git configuration.

Rename FileStoreAttributeCache to FileStoreAttributes since this class
is now declared public in order to enable exposing all attributes in one
object.

Example:

[filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"]
	timestampResolution = 7000 nanoseconds
	minRacyThreshold = 3440 microseconds

Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
2019-08-06 14:54:39 +02:00