Commit Graph

4915 Commits

Author SHA1 Message Date
Matthias Sohn e9fcf1932e JGit v5.3.3.201908210735-r
Change-Id: I799859e47616af0ac5ee3a982cb9350d141681d7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 13:30:43 +02:00
Matthias Sohn 48498fd3e6 Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Add missing @since tag on FileTreeIterator#getLastModifiedInstant
  Prepare 5.1.10-SNAPSHOT builds
  JGit v5.1.9.201908210455-r
  Avoid sign extension when comparing mtime with Instant#getEpochSecond
  Fix deprecation in DirCache caused by Instant based DirCacheEntry

Change-Id: If6d5f4dfd9fc8e8c09e29aa11b1004057eafeb9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 12:13:27 +02:00
Matthias Sohn 7ac3a63e7f Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Add missing @since tag on FileTreeIterator#getLastModifiedInstant
  Prepare 5.1.10-SNAPSHOT builds
  JGit v5.1.9.201908210455-r
  Avoid sign extension when comparing mtime with Instant#getEpochSecond
  Fix deprecation in DirCache caused by Instant based DirCacheEntry

Change-Id: Id824c0b8b14dad5947ae9da1f90c3471e07b400f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 12:02:33 +02:00
Matthias Sohn e80a62e226 Add missing @since tag on FileTreeIterator#getLastModifiedInstant
Change-Id: I809399e6a71e0079d2f0007b0d3f00b531d451bb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 11:52:20 +02:00
Matthias Sohn edc67125b7 Prepare 5.1.10-SNAPSHOT builds
Change-Id: If10afc8e663299a15db8c5fd0574fb51bf7e7ae9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 11:51:41 +02:00
Matthias Sohn 38adb548c3 JGit v5.1.9.201908210455-r
Change-Id: Iaade207292acb8b27e01aca7e1af97ad7db1e854
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 10:52:10 +02:00
Matthias Sohn c130e5e708 Avoid sign extension when comparing mtime with Instant#getEpochSecond
Ensure we use the same type when comparing seconds since the epoch.

This does not prevent that in 2038 timestamps in seconds since the epoch
stored in a 32 bit integer will overflow. Integer.MAX_VALUE translates
to 2038-01-19T03:14:07Z. After this date we'll have an issue since we
store seconds since the epoch in a 32 bit integer in some places.

Bug: 319142
Change-Id: If0c03003d40b480f044686e2f7a2f62c9f4e2fe1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 00:29:45 +02:00
Michael Keppler 6cb21049d7 Fix deprecation in DirCache caused by Instant based DirCacheEntry
Replace the two int variables smudge_s and smudge_ns by an Instant and
use the new method DirCacheEntry.mightBeRacilyClean(Instant).

Change-Id: Id70adbb0856a64909617acf65da1bae8e2ae934a
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-21 00:07:49 +02:00
Matthias Sohn 18fb58ae05 Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Fix NPE in RebaseTodoFile#parseComments
  Fix NPE in ObjectIdOwnerMap#get
  Fix NPE in CommitOnlyTest#getHead
  FileUtils#lastModifiedInstant should not log error if path doesn't exist
  Cache user global and system-wide git configurations
  Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
  Add missing dependencies for running FS_POSIXTest in Eclipse
  Fix javadoc for SystemReader#getInstance
  Improve retry handling when saving FileStoreAttributes fails
  Ensure FSTest uses MockSystemReader
  Make supportsAtomicCreateNewFile return true as default
  Update orbit to R20190602212107-2019-06 to enable backports from master
  Handle InvalidPathException in FS_POSIX#createNewFileAtomic
  Ensure root cause of lock creation failures is logged
  Implement toString in MockSystemReader and MockConfig
  LocalDiskRefTreeDatabaseTest shall use MockSystemReader
  Ensure LocalDiskRepositoryTestCase#setup fully uses MockSystemReader
  Ensure we use MockSystemReader in tests
  Override FileBasedConfig's save method in MockConfig
  Remove FileBasedConfig.load(boolean) introduced in d45219ba
  Disable debug log for FS in org.eclipse.jgit.test
  Bazel: enable logging for tests in org.eclipse.jgit.test
  LockFile: log exception if creation of lock file failed
  Stop using deprecated Constants.CHARACTER_ENCODING

Change-Id: If0c5010a2cf151ebebb2f2088fac3ee02c5007b9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-19 23:41:17 +02:00
Matthias Sohn 0eea7368e2 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix NPE in RebaseTodoFile#parseComments
  Fix NPE in ObjectIdOwnerMap#get
  Fix NPE in CommitOnlyTest#getHead
  FileUtils#lastModifiedInstant should not log error if path doesn't exist
  Cache user global and system-wide git configurations
  Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
  Add missing dependencies for running FS_POSIXTest in Eclipse
  Fix javadoc for SystemReader#getInstance
  Improve retry handling when saving FileStoreAttributes fails
  Ensure FSTest uses MockSystemReader
  Make supportsAtomicCreateNewFile return true as default
  Update orbit to R20190602212107-2019-06 to enable backports from master
  Handle InvalidPathException in FS_POSIX#createNewFileAtomic
  Ensure root cause of lock creation failures is logged
  Implement toString in MockSystemReader and MockConfig
  LocalDiskRefTreeDatabaseTest shall use MockSystemReader
  Ensure LocalDiskRepositoryTestCase#setup fully uses MockSystemReader
  Ensure we use MockSystemReader in tests
  Override FileBasedConfig's save method in MockConfig
  Remove FileBasedConfig.load(boolean) introduced in d45219ba
  Disable debug log for FS in org.eclipse.jgit.test
  Bazel: enable logging for tests in org.eclipse.jgit.test
  LockFile: log exception if creation of lock file failed
  Stop using deprecated Constants.CHARACTER_ENCODING

Change-Id: I48c585f3c9287be7d6ddb6b01a1955444e13fa31
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-19 18:30:35 +02:00
Matthias Sohn 86a567f615 Fix NPE in RebaseTodoFile#parseComments
Change-Id: I5487f3c2609eaf2a0ddf71ebb2f6c9701fb7600c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-19 10:16:23 +02:00
Matthias Sohn a4216e5382 Fix NPE in ObjectIdOwnerMap#get
Change-Id: I3812961a27ac410d610ef50c73a28f21bb05ae79
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-18 11:59:54 +02:00
Matthias Sohn 31356f5d18 FileUtils#lastModifiedInstant should not log error if path doesn't exist
Change-Id: Id8447735beb24becb41612d3d29d5351f8273d22
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-18 11:59:54 +02:00
Matthias Sohn f383206ace Cache user global and system-wide git configurations
So far the git configuration and the system wide git configuration were
always reloaded when jgit accessed these global configuration files to
access global configuration options which are not in the context of a
single git repository. Cache these configurations in SystemReader and
only reload them if their file metadata observed using FileSnapshot
indicates a modification.

Change-Id: I092fe11a5d95f1c5799273cacfc7a415d0b7786c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-08-18 11:47:26 +02:00
Matthias Sohn b51bb4f7bd Fix javadoc for SystemReader#getInstance
The existing javadoc was copied from another method and not adapted.

Change-Id: I39a7e5d719b2c379de9bd1a4710a55a73700c6f0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-16 16:51:45 +02:00
Matthias Sohn 2d84bb4341 Improve retry handling when saving FileStoreAttributes fails
- fix handling of interrupts in FileStoreAttributes#saveToConfig
- increase retry wait time to 100ms
- don't wait after last retry
- dont retry if failure is caused by another exception than
LockFailedException

Change-Id: I108c012717d2bcce71f2c6cb9cf0879de704ebc2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-13 08:33:32 +02:00
Vishal Devgire 9b4a55c0b6 Make supportsAtomicCreateNewFile return true as default
The method org.eclipse.jgit.util.FS.supportsAtomicCreateNewFile()
should default to true as mentioned in docs [1]

org.eclipse.jgit.util.FS_POSIX.supportsAtomicCreateNewFile() method
will set the value to false if the git config
core.supportsatomiccreatenewfile is not set.

It should default to true if the configuration is undefined.

[1]
4169a95a65/org.eclipse.jgit/src/org/eclipse/jgit/util/FS_POSIX.java (L372)

Bug: 544164
Change-Id: I16ccf989a89da2cf4975c200b3228b25ba4c0d55
Signed-off-by: Vishal Devgire <vishaldevgire@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-12 14:35:06 +02:00
Matthias Sohn 6ef225c519 Handle InvalidPathException in FS_POSIX#createNewFileAtomic
Bug: 547400
Change-Id: Ic3541e360a2968ba3532a3d3fa4828b0d0463c02
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-12 12:18:16 +02:00
Matthias Sohn cc29da8e0f Ensure root cause of lock creation failures is logged
Change-Id: I91cdf1e085a29c0aabd6d22c6ebe848b2d75f42c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-12 12:17:11 +02:00
Matthias Sohn a7338dd1e5 Remove FileBasedConfig.load(boolean) introduced in d45219ba
We can't add this method to the super class StoredConfig since that
abstracts from filesystem storage. MockSystemReader.MockConfig is a
StoredConfig and is also used by tests for dfs based storage. Hence
remove this leaky abstraction.

This implies we always use the fallback FileStoreAttributes which means
a config file modification is considered racy within the first 2
seconds. This should not be an issue since typically configs change
rarely and re-reading a config within the racy period is relatively
cheap since configs are small.

Change-Id: Ia2615addc24a7cadf3c566ee842c6f4f07e159a5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 23:57:21 +02:00
Matthias Sohn 42000a71e3 LockFile: log exception if creation of lock file failed
Change-Id: I4de75d12ec9e61193494916307289378cdb6220e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-10 00:57:04 +02:00
David Pursehouse ad02a7cbad Stop using deprecated Constants.CHARACTER_ENCODING
Change-Id: I105b8a05bc64f249879a0795a059958553cc60c6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-09 12:11:41 +09:00
Matthias Sohn 718555518d Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of
StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: I238adfd3080a5fed9d64c3c757297da6ea893918
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 13:04:38 +02:00
Matthias Sohn 0046b2a8fe Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: Ia57385b2a60f48a5317c8d723721c235d7043a84
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 11:54:56 +02:00
Han-Wen Nienhuys e60b9e1879 FileSnapshot: fix bug with timestamp thresholding
Increase the safety factor to 2.5x for extra safety if max of measured
timestamp resolution and measured minimal racy threshold is < 100ms, use
1.25 otherwise since for large filesystem resolution values the
influence of finite resolution of the system clock should be negligible.

Before, not yet using the newly introduced minRacyThreshold measurement,
the threshold was 1.1x FS resolution, and we could issue the
following sequence of events,

  start
  create-file
  read-file (currentTime)
  end

which had the following timestamps:

  create-file 1564589081998
  start 1564589082002
  read 1564589082003
  end 1564589082004

In this case, the difference between create-file and read is 5ms,
which exceeded the 4ms FS resolution, even though the events together
took just 2ms of runtime.

Reproduce with:
  bazel test --runs_per_test=100 \
    //org.eclipse.jgit.test:org_eclipse_jgit_internal_storage_file_FileSnapshotTest

The file system timestamp resolution is 4ms in this case.

This code assumes that the kernel and the JVM use the same clock that
is synchronized with the file system clock. This seems plausible,
given the resolution of System.currentTimeMillis() and the latency for
a gettimeofday system call (typically ~1us), but it would be good to
justify this with specifications.

Also cover a source of flakiness: if the test runs under extreme load,
then we could have

  start
  create-file
  <long delay>
  read
  end

which would register as an unmodified file. Avoid this by skipping the
test if end-start is too big.

[msohn]:
- downported from master to stable-5.1
- skip test if resolution is below 10ms
- adjust safety factor to 1.25 for resolutions above 100ms

Change-Id: I87d2cf035e01c44b7ba8364c410a860aa8e312ef
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-07 13:56:08 +02:00
Matthias Sohn 3b368d5578 In LockFile#waitForStatChange wait in units of file time resolution
Since we now measure file time resolution we can use it to replace the
hard coded wait time of 25ms. FileSnapshot#equals will return true until
the mtime of the old (o) and the new FileSnapshot (n) differ by at least
one  file time resolution.

Change-Id: Icb713a80ce9eb929242ed083406bfb6650c72223
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-07 13:21:39 +02:00
Matthias Sohn 6857138e19 Cache FileStoreAttributeCache per directory
Cache FileStoreAttributeCache entries since looking up FileStore for a
file may be expensive on some platforms.

Implement a simple LRU cache based on ConcurrentHashMap using a simple
long counter to order access to cache entries.

Change-Id: I4881fa938ad2f17712c05da857838073a2fc4ddb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
2019-08-06 14:54:39 +02:00
Matthias Sohn 275f3da783 Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
Use the fallback timestamp resolution as already described in the
javadoc of these methods. Using zero file timestamp resolution doesn't
make sense.

Change-Id: Iaad2a0f99c3be3678e94980a0a368181b6aed38c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-06 14:54:39 +02:00
Matthias Sohn d45219baac Persist minimal racy threshold and allow manual configuration
To enable persisting the minimal racy threshold per FileStore add a
new config option to the user global git configuration:

- Config section is "filesystem"
- Config subsection is concatenation of
  - Java vendor (system property "java.vendor")
  - Java version (system property "java.version")
  - FileStore's name, on Windows we use the attribute volume:vsn instead
    since  the name is not necessarily unique.
  - separated by '|'
  e.g.
    "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
  The same prefix is used as for filesystem timestamp resolution, so
  both values are stored in the same config section
- The config key for minmal racy threshold is "minRacyThreshold" as a
  time value, supported time units are those supported by
  DefaultTypedConfigGetter#getTimeUnit
- measure for 3 seconds to limit runtime which depends on hardware, OS
  and Java version being used

If the minimal racy threshold is configured for a given FileStore the
configured value is used instead of measuring it.

When the minimal racy threshold was measured it is persisted in the user
global git configuration.

Rename FileStoreAttributeCache to FileStoreAttributes since this class
is now declared public in order to enable exposing all attributes in one
object.

Example:

[filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"]
	timestampResolution = 7000 nanoseconds
	minRacyThreshold = 3440 microseconds

Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
2019-08-06 14:54:39 +02:00
Matthias Sohn 5911521ba6 Measure minimum racy interval to auto-configure FileSnapshot
By running FileSnapshotTest#detectFileModified we found that the sum of
measured filesystem timestamp resolution and measured clock resolution
may yield a too small interval after a file has been modified which we
need to consider racily clean. In our tests we didn't find this behavior
on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11
this effect was not observed.

On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the
following test results using Java 8 and 11:

In 23-98% of 10000 test runs (depending on filesystem type and Java
version) the test failed, which means the effective interval which needs
to be considered racily clean after a file was modified is larger than
the measured file timestamp resolution.

"delta" is the observed interval after a file has been modified but
FileSnapshot did not yet detect the modification:

"resolution" is the measured sum of file timestamp resolution and clock
resolution seen in Java.

Java version    filesystem  failures resolution  min delta   max delta
1.8.0_212-b04   btrfs	    98.6%       1 ms        3.6 ms      6.6 ms
1.8.0_212-b04   ext4        82.6%       3 ms        1.1 ms      4.1 ms
1.8.0_212-b04   xfs         23.8%       4 ms        3.7 ms      3.9 ms
1.8.0_212-b04   zfs         23.1%       3 ms        4.8 ms      5.0 ms
11.0.3+7        btrfs       98.1%       3 us        0.7 ms      4.7 ms
11.0.3+7        ext4        98.1%       6 us        0.7 ms      4.7 ms
11.0.3+7        xfs         98.5%       7 us        0.1 ms      8.0 ms
11.0.3+7        zfs         98.4%       7 us        0.7 ms      5.2 ms

Mac OS
1.8.0_212       APFS        0%          1 s
11.0.3+7        APFS        0%          6 us

The observed delta is not distributed according to a normal gaussian
distribution but rather random in the observed range between "min delta"
and "max delta".

Run this test after measuring file timestamp resolution in
FS.FileAttributeCache to auto-configure JGit since it's unclear what
mechanism is causing this effect.

In FileSnapshot#isRacyClean use the maximum of the measured timestamp
resolution and the measured "delta" as explained above to decide if a
given FileSnapshot is to be considered racily clean. Add a 30% safety
margin to ensure we are on the safe side.

Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-06 14:54:35 +02:00
Matthias Sohn d17efe880d Fix FileAttributeCache.toString()
We should not list the complete cache but only show the cache entry at
hand.

Change-Id: I22be2a4dcbf0145155e23f2389bfcf5662cf23a6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-26 15:53:29 +02:00
Matthias Sohn 130aa31262 Add test for racy git detection in FileSnapshot
Repeat the test 10000 times to get statistics if measured
fsTimestampResolution is working in practice to detect racy git
situations.

Add a class to compute statistics for this test. Log delta between
lastModified and time when FileSnapshot failed to detect modification.
This happens if the racy git limit determined by measuring filesystem
timestamp resolution and clock resolution is too small. If it would be
correct FileSnapshot would always detect modification or mark it
modified if time since modification is smaller than the racy git limit.

Change-Id: Iabe7af1a7211ca58480f8902d4fa4e366932fc77
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-26 15:49:38 +02:00
Matthias Sohn 99d351d0cb Measure stored timestamp resolution instead of time to touch file
Measure granularity of timestamps stored in the filesystem by setting
and then getting lastModified timestamp until the read value changed.
Increase increment exponentially to limit number of iterations starting
with 1 microsecond since Java's FileTime (up to Java 12) truncates
timestamps to 1 microsecond resolution. The chosen algorithm yields 2000
steps between 1 ms and 2.5 s.

Also measure clock resolution and add that for the total timestamp
resolution. This avoids systematic measurement errors introduced by
doing IO to touch a file.

Change-Id: I9b37138619422452373e298d9d8c7cb2c384db3f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-19 14:45:55 +02:00
Matthias Sohn 37f7679fc9 Handle CancellationException in FileStoreAttributeCache
Change-Id: If5985fbf04f630b1d72a1bafd508e0e15e1436be
2019-07-19 14:45:55 +02:00
Matthias Sohn 72d6e304b8 Fix FileSnapshot#saveNoConfig
We should not use configuration when creating FileSnapshot when
accessing FileBasedConfig.

Change-Id: Ic521632870f18bb004751642b9d30648dd94049a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-19 14:45:55 +02:00
Matthias Sohn d8d9427277 Use Instant for smudge time in DirCache and DirCacheEntry
Change-Id: I98050a51baf4726c5717ef62ce7f026173666bdf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-18 03:36:18 +02:00
Matthias Sohn 95e8264cc8 Use Instant instead of milliseconds for filesystem timestamp handling
This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.

Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution

Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-18 03:27:52 +02:00
Matthias Sohn 4db39f5074 Workaround SecurityException in FS#getFsTimestampResolution
On Android FS#getFsTimestampResolution always throws a
SecurityException, handle this by falling back to the fallback timestamp
resolution.

Bug: 548947
Change-Id: I0ee6cb3c20e189bdc8d488434a930427ad6f2df2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:15 +02:00
Matthias Sohn 79ede0c322 Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
Bug: 548682
Change-Id: I48840d3a68cf1db92c056d218a0d5ed0b9ea4c45
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:14 +02:00
Matthias Sohn be66222d1a FS: ignore AccessDeniedException when measuring timestamp resolution
It seems on cygwin creating a file under a writable directory can fail
with AccessDeniedException. Log a warning in this case and fallback to
worst case timestamp resolution of 2 seconds.

Bug: 548648
Change-Id: Ic50c31ce9dc9ccadd4db5247df929418ac62d45c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:14 +02:00
Matthias Sohn 09fec1a102 Add debug trace for FileSnapshot
Checking lastModified is time critical hence debug trace is the only way
to analyze issues since debugging is impractical.

Also add configuration for buffering of log4j output to reduce runtime
impact when debug trace is on. Limit buffer to 1MiB and comment this
configuration out since we may not always want to use buffering.

Change-Id: Ib1a0537b67c8dc3fac994a77b42badd974ce6c97
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:14 +02:00
Matthias Sohn 8a0ed8a004 Use FileChannel.open to touch file and set mtime to now
Use options
- StandardOpenOption.CREATE to create touched file if not existing
- StandardOpenOption.SYNC to enforce synch of data and meta data changes
- StandardOpenOption.WRITE

Also set mtime explicitly in FileUtils#touch to the current system time.
This should fix that the previous implementation didn't work on
- locally cached Windows network share (CSC-CACHE filesystem) mapped as
a drive
- nfsv4 mounts on Linux
and that it didn't create unborn file like Linux command "touch".
Apache common's and Guava's touch() use the same approach.

Immediately after creating the probe file used to measure timestamp
resolution touch it. This ensures we always use the local system clock
when measuring filesystem timestamp resolution. This should prevent that
clock skew could influence the measured timestamp resolution in case of
a mounted network filesystem.

Bug: 548598
Change-Id: Iaeaf5967963f582395a195aa637b8188bfadac60
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:14 +02:00
Matthias Sohn 16760c3e9a Persist filesystem timestamp resolution and allow manual configuration
To enable persisting filesystem timestamp resolution per FileStore add a
new config section to the user global git configuration:

- Config section is "filesystem"
- Config subsection is concatenation of
  - Java vendor (system property "java.vm.vendor")
  - runtime version (system property "java.vm.version")
  - FileStore's name
  - separated by '|'
  e.g.
    "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
  The prefix is needed since some Java versions do not expose the full
  timestamp resolution of the underlying filesystem. This may also
  depend on the underlying operating system hence concrete key values
  may not be portable.
- Config key for timestamp resolution is "timestampResolution" as a time
  value, supported time units are those supported by
  DefaultTypedConfigGetter#getTimeUnit

If timestamp resolution is already configured for a given FileStore
the configured value is used instead of measuring the resolution.

When timestamp resolution was measured it is persisted in the user
global git configuration.

Example:

[filesystem "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"]
	timestampResolution = 1 seconds

If locking the git config file fails retry saving the resolution up to 5
times in order to workaround races with another thread.

In order to avoid stack overflow use the fallback filesystem timestamp
resolution when loading FileBasedConfig which creates itself a
FileSnapshot to help checking if the config changed.

Note:
- on some OSes Java 8,9 truncate to milliseconds or seconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes up to Java 12 truncates timestamp resolution to
microseconds when converting the internal representation to FileTime
exposed in the API, see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution up to
Java 12

Hence do not attempt to manually configure a higher timestamp resolution
than supported by the Java version being used at runtime.

Bug: 546891
Bug: 548188
Change-Id: Iff91b8f9e6e5e2295e1463f87c8e95edf4abbcf8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:14 +02:00
Matthias Sohn 121c957405 Add support for nanoseconds and microseconds for Config#getTimeUnit
Change-Id: I0a5828438810dd23790cba52d7ae2e055c6a3fc9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-11 10:12:04 +02:00
Matthias Sohn a950eac23b Optionally measure filesystem timestamp resolution asynchronously
In order to avoid blocking on the main thread during measurement
interactive applications like EGit may want to measure the filesystem
timestamp resolution asynchronously.

In order to enable measurement in the background call
FileStoreAttributeCache.setAsyncfileStoreAttrCache(true)
before the first access to cached FileStore attributes.

Bug: 548188
Change-Id: I8c9a2dbfc3f1d33441edea18b90e36b1dc0156c7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-11 10:12:04 +02:00
Han-Wen Nienhuys 84e6c24e58 FileSnapshot#equals: consider UNKNOWN_SIZE
Add a unittest.

In commit I5485db55 ("Fix FileSnapshot's consideration of file size"),
the special casing of UNKNOWN_SIZE was forgotten.

This change, together with I493f3b57b ("Measure file timestamp
resolution used in FileSnapshot") introduced a regression that would
occasionally surface in Gerrit integration tests marked UseLocalDisk,
with the symptom that creating the Admin user in NoteDb failed with a
LOCK_FAILURE.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I7ffd972581f815c144f810481103c7985af5feb0
2019-07-09 15:33:23 +02:00
Matthias Sohn 850b9d7540 Timeout measuring file timestamp resolution after 2 seconds
It was reported that measuring file timestamp resolution may hang
indefinitely on nfs. Hence timeout this measurement at the known worst
filesystem timestamp resolution (FAT) of 2 seconds.

Bug: 548188
Change-Id: I17004b0aa49d5b0e76360a008af3adb911b289c0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-03 21:34:29 +02:00
David Pursehouse f18b5010fc Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
Change-Id: I621ba174235a6fb56236e54d24bce704bb5afb28
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-19 10:43:01 +02:00
Matthias Sohn 9387288a86 Fix non-deterministic hash of archives created by ArchiveCommand
Archives created by the ArchiveCommand didn't produce deterministic
archive hashes. For RevCommits RevWalk.parseTree returns the root tree
instead of the RevCommit hence retrieving the commit's timestamp didn't
work. Instead use RevWalk.parseAny and extract the tree manually.

Archive entries store timestamps with 1 second resolution hence we need
to wait longer when creating the same archive twice and compare archive
hashes. Otherwise hash comparison in tests wouldn't fail without this
patch.

Bug: 548312
Change-Id: I437d515de51cf68265584d28a8446cebe6341b79
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-19 10:43:00 +02:00
Matthias Sohn 2d0a1adf05 Update Maven plugins and cleanup Maven warnings
update Maven plugins
- jacoco-maven-plugin to 0.8.4
- japicmp-maven-plugin to 0.14.1
- maven-compiler-plugin to 3.8.1
- maven-deploy-plugin to 3.0.0-M1
- maven-enforcer-plugin to 3.0.0-M2
- maven-install-plugin to 3.0.0-M1
- maven-jar-plugin to 3.1.2
- maven-javadoc-plugin to 3.1.0
- maven-jxr-plugin to 3.0.0
- maven-pmd-plugin to 3.12.0
- maven-resources-plugin to 3.1.0
- maven-shade-plugin to 3.2.1
- maven-source-plugin to 3.1.0
- maven-surefire-plugin to 3.0.0-M3
- spotbugs-maven-plugin to 3.1.12
- tycho to 1.3.0
- tycho-pack200a-plugin to 1.3.0
- tycho-pack200b-plugin to 1.3.0

Cleanup Maven warnings
- pin version of all used Maven plugins
- remove deprecated way to declare minimum Maven version

Change-Id: If23e2e2bb03e5e1e7b1eb9d4924a8faa0aa3704e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-18 16:55:44 +02:00