Commit Graph

2505 Commits

Author SHA1 Message Date
David Pursehouse 72ae089206 Fix NarrowingCompoundAssignment warnings from Error Prone
Error Prone reports:

  [NarrowingCompoundAssignment] Compound assignments from long to int
  hide lossy casts

and

  [NarrowingCompoundAssignment] Compound assignments from int to byte
  hide lossy casts

  See https://errorprone.info/bugpattern/NarrowingCompoundAssignment

Fix the warnings by adding explicit casts or changing types as
necessary.

Now that all occurrences of the warning are fixed, increase its
severity to ERROR.

Change-Id: Idb3670e6047b146ae37daee07212ff9455512623
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-08-08 14:28:27 +02:00
Matthias Sohn f54db4a857 Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: Iec3ad6ccc194582cb844310dc172c3103dae4457
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 13:24:14 +02:00
Matthias Sohn 718555518d Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of
StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: I238adfd3080a5fed9d64c3c757297da6ea893918
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 13:04:38 +02:00
Matthias Sohn 0046b2a8fe Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix OpenSshConfigTest#config
  FileSnapshot: fix bug with timestamp thresholding
  In LockFile#waitForStatChange wait in units of file time resolution
  Cache FileStoreAttributeCache per directory
  Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
  Persist minimal racy threshold and allow manual configuration
  Measure minimum racy interval to auto-configure FileSnapshot
  Reuse FileUtils to recursively delete files created by tests
  Fix FileAttributeCache.toString()
  Add test for racy git detection in FileSnapshot
  Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
  Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
  Add missing javadoc in org.eclipse.jgit.junit
  Enhance RepeatRule to report number of failures at the end
  Fix FileSnapshotTests for filesystem with high timestamp resolution
  Retry deleting test files in FileBasedConfigTest
  Measure filesystem timestamp resolution already in test setup
  Refactor FileSnapshotTest to use NIO APIs
  Measure stored timestamp resolution instead of time to touch file
  Handle CancellationException in FileStoreAttributeCache
  Fix FileSnapshot#saveNoConfig
  Use Instant for smudge time in DirCache and DirCacheEntry
  Use Instant instead of milliseconds for filesystem timestamp handling
  Workaround SecurityException in FS#getFsTimestampResolution
  Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
  FS: ignore AccessDeniedException when measuring timestamp resolution
  Add debug trace for FileSnapshot
  Use FileChannel.open to touch file and set mtime to now
  Persist filesystem timestamp resolution and allow manual configuration
  Increase bazel timeout for long running tests
  Bazel: Fix lint warning flagged by buildifier
  Update bazlets to latest version
  Bazel: Add missing dependencies for ArchiveCommandTest
  Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
  Add support for nanoseconds and microseconds for Config#getTimeUnit
  Optionally measure filesystem timestamp resolution asynchronously
  Delete unused FileTreeIteratorWithTimeControl
  FileSnapshot#equals: consider UNKNOWN_SIZE
  Timeout measuring file timestamp resolution after 2 seconds
  Fix RacyGitTests#testRacyGitDetection
  Change RacyGitTests to create a racy git situation in a stable way
  Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
  Fix non-deterministic hash of archives created by ArchiveCommand
  Update Maven plugins ecj, plexus, error-prone
  Update Maven plugins and cleanup Maven warnings
  Make inner classes static where possible
  Fix API problem filters

Change-Id: Ia57385b2a60f48a5317c8d723721c235d7043a84
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 11:54:56 +02:00
Matthias Sohn 5a88815b1c Fix OpenSshConfigTest#config
- use FS.DETECTED instead of db.getFS() since the ssh config is
typically in a different place than the repository, the same is used in
OpenSshConfig
- reduce unnecessary repeated writes by introducing wait for one tick of
the file time resolution

Change-Id: Ifac915e97ff420ec5cf8e2f162e351f9f51b6b14
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-08 10:10:12 +02:00
Han-Wen Nienhuys e60b9e1879 FileSnapshot: fix bug with timestamp thresholding
Increase the safety factor to 2.5x for extra safety if max of measured
timestamp resolution and measured minimal racy threshold is < 100ms, use
1.25 otherwise since for large filesystem resolution values the
influence of finite resolution of the system clock should be negligible.

Before, not yet using the newly introduced minRacyThreshold measurement,
the threshold was 1.1x FS resolution, and we could issue the
following sequence of events,

  start
  create-file
  read-file (currentTime)
  end

which had the following timestamps:

  create-file 1564589081998
  start 1564589082002
  read 1564589082003
  end 1564589082004

In this case, the difference between create-file and read is 5ms,
which exceeded the 4ms FS resolution, even though the events together
took just 2ms of runtime.

Reproduce with:
  bazel test --runs_per_test=100 \
    //org.eclipse.jgit.test:org_eclipse_jgit_internal_storage_file_FileSnapshotTest

The file system timestamp resolution is 4ms in this case.

This code assumes that the kernel and the JVM use the same clock that
is synchronized with the file system clock. This seems plausible,
given the resolution of System.currentTimeMillis() and the latency for
a gettimeofday system call (typically ~1us), but it would be good to
justify this with specifications.

Also cover a source of flakiness: if the test runs under extreme load,
then we could have

  start
  create-file
  <long delay>
  read
  end

which would register as an unmodified file. Avoid this by skipping the
test if end-start is too big.

[msohn]:
- downported from master to stable-5.1
- skip test if resolution is below 10ms
- adjust safety factor to 1.25 for resolutions above 100ms

Change-Id: I87d2cf035e01c44b7ba8364c410a860aa8e312ef
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-07 13:56:08 +02:00
Matthias Sohn 6857138e19 Cache FileStoreAttributeCache per directory
Cache FileStoreAttributeCache entries since looking up FileStore for a
file may be expensive on some platforms.

Implement a simple LRU cache based on ConcurrentHashMap using a simple
long counter to order access to cache entries.

Change-Id: I4881fa938ad2f17712c05da857838073a2fc4ddb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
2019-08-06 14:54:39 +02:00
Matthias Sohn d45219baac Persist minimal racy threshold and allow manual configuration
To enable persisting the minimal racy threshold per FileStore add a
new config option to the user global git configuration:

- Config section is "filesystem"
- Config subsection is concatenation of
  - Java vendor (system property "java.vendor")
  - Java version (system property "java.version")
  - FileStore's name, on Windows we use the attribute volume:vsn instead
    since  the name is not necessarily unique.
  - separated by '|'
  e.g.
    "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
  The same prefix is used as for filesystem timestamp resolution, so
  both values are stored in the same config section
- The config key for minmal racy threshold is "minRacyThreshold" as a
  time value, supported time units are those supported by
  DefaultTypedConfigGetter#getTimeUnit
- measure for 3 seconds to limit runtime which depends on hardware, OS
  and Java version being used

If the minimal racy threshold is configured for a given FileStore the
configured value is used instead of measuring it.

When the minimal racy threshold was measured it is persisted in the user
global git configuration.

Rename FileStoreAttributeCache to FileStoreAttributes since this class
is now declared public in order to enable exposing all attributes in one
object.

Example:

[filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"]
	timestampResolution = 7000 nanoseconds
	minRacyThreshold = 3440 microseconds

Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
2019-08-06 14:54:39 +02:00
Matthias Sohn 5911521ba6 Measure minimum racy interval to auto-configure FileSnapshot
By running FileSnapshotTest#detectFileModified we found that the sum of
measured filesystem timestamp resolution and measured clock resolution
may yield a too small interval after a file has been modified which we
need to consider racily clean. In our tests we didn't find this behavior
on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11
this effect was not observed.

On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the
following test results using Java 8 and 11:

In 23-98% of 10000 test runs (depending on filesystem type and Java
version) the test failed, which means the effective interval which needs
to be considered racily clean after a file was modified is larger than
the measured file timestamp resolution.

"delta" is the observed interval after a file has been modified but
FileSnapshot did not yet detect the modification:

"resolution" is the measured sum of file timestamp resolution and clock
resolution seen in Java.

Java version    filesystem  failures resolution  min delta   max delta
1.8.0_212-b04   btrfs	    98.6%       1 ms        3.6 ms      6.6 ms
1.8.0_212-b04   ext4        82.6%       3 ms        1.1 ms      4.1 ms
1.8.0_212-b04   xfs         23.8%       4 ms        3.7 ms      3.9 ms
1.8.0_212-b04   zfs         23.1%       3 ms        4.8 ms      5.0 ms
11.0.3+7        btrfs       98.1%       3 us        0.7 ms      4.7 ms
11.0.3+7        ext4        98.1%       6 us        0.7 ms      4.7 ms
11.0.3+7        xfs         98.5%       7 us        0.1 ms      8.0 ms
11.0.3+7        zfs         98.4%       7 us        0.7 ms      5.2 ms

Mac OS
1.8.0_212       APFS        0%          1 s
11.0.3+7        APFS        0%          6 us

The observed delta is not distributed according to a normal gaussian
distribution but rather random in the observed range between "min delta"
and "max delta".

Run this test after measuring file timestamp resolution in
FS.FileAttributeCache to auto-configure JGit since it's unclear what
mechanism is causing this effect.

In FileSnapshot#isRacyClean use the maximum of the measured timestamp
resolution and the measured "delta" as explained above to decide if a
given FileSnapshot is to be considered racily clean. Add a 30% safety
margin to ensure we are on the safe side.

Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-08-06 14:54:35 +02:00
Matthias Sohn 130aa31262 Add test for racy git detection in FileSnapshot
Repeat the test 10000 times to get statistics if measured
fsTimestampResolution is working in practice to detect racy git
situations.

Add a class to compute statistics for this test. Log delta between
lastModified and time when FileSnapshot failed to detect modification.
This happens if the racy git limit determined by measuring filesystem
timestamp resolution and clock resolution is too small. If it would be
correct FileSnapshot would always detect modification or mark it
modified if time since modification is smaller than the racy git limit.

Change-Id: Iabe7af1a7211ca58480f8902d4fa4e366932fc77
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-26 15:49:38 +02:00
Matthias Sohn de6df3bdd9 Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
This should help to detect if measured fsTimeResolution is too small.

Change-Id: Id1f54dbdedb52b17859904e47776fa3a5887b8be
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-24 16:21:54 +02:00
Matthias Sohn 9eff45e4f2 Fix FileSnapshotTests for filesystem with high timestamp resolution
When filesystem timestamp resolution is very high some tests don't work
since runtime of the test setup is too long to reach a racily clean
FileSnapshot. Hence skip these tests when timestamp resolution is higher
than 10 millisecond.

Change-Id: Ie47dd10eda22037b5c1ebff6b6becce0654ea807
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-19 14:45:56 +02:00
Matthias Sohn bce4ac97fa Retry deleting test files in FileBasedConfigTest
Change-Id: I304b2b6f2e39f72f620bba53aead60256aed3660
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-19 14:45:55 +02:00
Matthias Sohn eda2e95fa8 Measure filesystem timestamp resolution already in test setup
This helps to avoid some time critical tests can't prepare the test
fixture intended since measuring timestamp resolution takes time.

Change-Id: Ib34023e682a106070ca97e98ef16789a4dfb97b4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-19 14:45:55 +02:00
Matthias Sohn 93144f1438 Refactor FileSnapshotTest to use NIO APIs
- use Path instead of File
- create test directories, files and output stream using Files methods
- delete unused list "files"

Change-Id: I8c5c601eca9f613efb5618d33b262277df92a06a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-19 14:45:55 +02:00
Matthias Sohn 95e8264cc8 Use Instant instead of milliseconds for filesystem timestamp handling
This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.

Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution

Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-18 03:27:52 +02:00
Matthias Sohn 79ede0c322 Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
Bug: 548682
Change-Id: I48840d3a68cf1db92c056d218a0d5ed0b9ea4c45
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:14 +02:00
Matthias Sohn 09fec1a102 Add debug trace for FileSnapshot
Checking lastModified is time critical hence debug trace is the only way
to analyze issues since debugging is impractical.

Also add configuration for buffering of log4j output to reduce runtime
impact when debug trace is on. Limit buffer to 1MiB and comment this
configuration out since we may not always want to use buffering.

Change-Id: Ib1a0537b67c8dc3fac994a77b42badd974ce6c97
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-17 09:42:14 +02:00
Matthias Sohn 0966731cad Increase bazel timeout for long running tests
EolRepositoryTest and GcCommitSelectionTest timed out frequently when
running unit tests using bazel with the default timeout "moderate"
(300s). Increase timeout of these tests to "long" (900s).

Change-Id: I43588cf950f55b50f868d9fe9c66d22bd428a54c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-16 12:11:22 +09:00
David Ostrovsky bbef67e8d0 Bazel: Fix lint warning flagged by buildifier
This change is fixing confusing name warning: [1].

  ./org.eclipse.jgit.test/tests.bzl:12: confusing-name:
  Never use 'l', 'I', or 'O' as names (they're too easily confused
  with 'I', 'l', or '0').

And is also fixing: "All calls to rules or macros should pass arguments
by keyword position argument" warning: [2].

  ./org.eclipse.jgit.test/BUILD:42: positional-args: All calls to rules
  or macros should pass arguments by keyword (arg_name=value) syntax.

[1] https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#confusing-name
[2] https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#positional-args

Change-Id: If5c28ec8a1ddc1d1b1035bd07b838a2a564aea4f
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2019-07-16 12:10:31 +09:00
David Pursehouse 0aa8eca5f1 Bazel: Add missing dependencies for ArchiveCommandTest
Dependencies on commons-compress, xz, and jgit-archive are required
for the build to succeed.

Change-Id: I42f3721078a240ad93b8dcab909e66b9bfff0b56
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-07-16 12:10:31 +09:00
David Pursehouse e54fde8616 Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
FileTreeIteratorWithTimeControl was deleted in a024759, but was
not removed from the BUILD file, thus causing the bazel build to
fail.

Change-Id: I892c0ffcac947298d0d6009374ee2c5d9afefb66
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-07-16 11:20:21 +09:00
Matthias Sohn 121c957405 Add support for nanoseconds and microseconds for Config#getTimeUnit
Change-Id: I0a5828438810dd23790cba52d7ae2e055c6a3fc9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-11 10:12:04 +02:00
Matthias Sohn a024759cf5 Delete unused FileTreeIteratorWithTimeControl
The only usage of this test iterator was removed in df637928d. Hence
delete this iterator and associated test.

Change-Id: I47710133ec3edc675c21db210960c024982668c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-07-11 10:00:23 +02:00
Han-Wen Nienhuys 84e6c24e58 FileSnapshot#equals: consider UNKNOWN_SIZE
Add a unittest.

In commit I5485db55 ("Fix FileSnapshot's consideration of file size"),
the special casing of UNKNOWN_SIZE was forgotten.

This change, together with I493f3b57b ("Measure file timestamp
resolution used in FileSnapshot") introduced a regression that would
occasionally surface in Gerrit integration tests marked UseLocalDisk,
with the symptom that creating the Admin user in NoteDb failed with a
LOCK_FAILURE.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I7ffd972581f815c144f810481103c7985af5feb0
2019-07-09 15:33:23 +02:00
Matthias Sohn 1159f9dd7c Fix RacyGitTests#testRacyGitDetection
This test case assumed file system timestamp resolution of 1 second. On
filesystems with a finer resolution this test fails since the index
entry is only smudged if the file index entry's lastModified and the
lastModified of the git index itself are within the same filesystem
timer tick. Fix this by ensuring that these timestamps are identical
which should work for any filesystem timer resolution.

Bug: 548188
Change-Id: Id84d59e1cfeb48fa008f8f27f2f892c4f73985de
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
2019-07-03 21:34:27 +02:00
Masaya Suzuki df637928d2 Change RacyGitTests to create a racy git situation in a stable way
By using File#setLastModified, we can create a racy git situation
stably.

Tested with --runs_per_test=100

Bug: 526111
Change-Id: I60b3632d353e19f335668325aa603640be423f58
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2019-06-19 10:43:03 +02:00
David Pursehouse f18b5010fc Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
Change-Id: I621ba174235a6fb56236e54d24bce704bb5afb28
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-19 10:43:01 +02:00
Matthias Sohn 9387288a86 Fix non-deterministic hash of archives created by ArchiveCommand
Archives created by the ArchiveCommand didn't produce deterministic
archive hashes. For RevCommits RevWalk.parseTree returns the root tree
instead of the RevCommit hence retrieving the commit's timestamp didn't
work. Instead use RevWalk.parseAny and extract the tree manually.

Archive entries store timestamps with 1 second resolution hence we need
to wait longer when creating the same archive twice and compare archive
hashes. Otherwise hash comparison in tests wouldn't fail without this
patch.

Bug: 548312
Change-Id: I437d515de51cf68265584d28a8446cebe6341b79
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-19 10:43:00 +02:00
Matthias Sohn 2d0a1adf05 Update Maven plugins and cleanup Maven warnings
update Maven plugins
- jacoco-maven-plugin to 0.8.4
- japicmp-maven-plugin to 0.14.1
- maven-compiler-plugin to 3.8.1
- maven-deploy-plugin to 3.0.0-M1
- maven-enforcer-plugin to 3.0.0-M2
- maven-install-plugin to 3.0.0-M1
- maven-jar-plugin to 3.1.2
- maven-javadoc-plugin to 3.1.0
- maven-jxr-plugin to 3.0.0
- maven-pmd-plugin to 3.12.0
- maven-resources-plugin to 3.1.0
- maven-shade-plugin to 3.2.1
- maven-source-plugin to 3.1.0
- maven-surefire-plugin to 3.0.0-M3
- spotbugs-maven-plugin to 3.1.12
- tycho to 1.3.0
- tycho-pack200a-plugin to 1.3.0
- tycho-pack200b-plugin to 1.3.0

Cleanup Maven warnings
- pin version of all used Maven plugins
- remove deprecated way to declare minimum Maven version

Change-Id: If23e2e2bb03e5e1e7b1eb9d4924a8faa0aa3704e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-18 16:55:44 +02:00
David Pursehouse 6d0168a414 Make inner classes static where possible
As reported by Error Prone:

An inner class should be static unless it references members of its
enclosing class. An inner class that is made non-static unnecessarily
uses more memory and does not make the intent of the class clear.

See https://errorprone.info/bugpattern/ClassCanBeStatic

Change-Id: Ib99d120532630dba63cf400cc1c61c318286fc41
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ee40efcea4)
2019-06-17 07:48:34 +02:00
David Pursehouse b84235c4d1 BatchRefUpdateTest: Suppress ImmutableEnumChecker warning
Change-Id: I0644690e93c19c224c1d896fe3593f8b38901cbc
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-17 08:18:12 +09:00
Matthias Sohn f3b7c2beae Prepare 5.4.1-SNAPSHOT builds
Change-Id: I8620d03c1cd5c1d0ad04e7607553e9aa18def9bb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-12 17:35:11 +02:00
Matthias Sohn 3dad0682be JGit v5.4.0.201906121030-r
Change-Id: Ie68828af68d07cf8e3fe778d39436f539f1c73d0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-12 16:29:11 +02:00
Matthias Sohn 0e9361f066 JGit v5.4.0.201906120450-r
Change-Id: Iea17cba848d4e53d69d34e952a2476c375721aa4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-12 10:50:14 +02:00
Matthias Sohn c177c6f4c5 Merge branch 'master' into stable-5.4
* master:
  Handle missing "ours" stage in WorkingTreeIterator.hasCrLfInIndex()
  Config: Add helper method to check for empty value
  ErrorProne: Increase severity of FutureReturnValueIgnored to ERROR
  FS_Win32: Add missing parentheses on if-blocks
  Upgrade spotbugs to 3.1.12
  Abbreviated{Long}ObjectId: Make operator precedence explicit
  GC: Update TODO comments
  FS_POSIX: Fix reference comparison of Boolean.FALSE
  Increase bazel timeout for long running tests
  Use bazelisk to switch between used bazel version
  Bump minimum Bazel version to 0.26.1
  Bazel: Bump skylib library version to 0.8.0
  Retry loading config when locked by another process
  Make pull --rebase on an unborn branch do a checkout
  Warn if configured cookie file is missing
  Handle escaped CR-LF in git config files
  DescribeCommand: use glob match instead of path match
  Fix off-by-one error in RebaseTodoFile when reading a todo file
  Consistently use "!isEmpty()" to detect non-empty list
  TransportHttp: Check for non-empty list with "!isEmpty()" rather than
    "size() > 0"
  TransportHttp: Fix comparison of size with ">= 0"
  NetscapeCookieFileTest: Split HttpCookiesMatcher to own class
  Bazel: Add missing dependency on mockito for TransportHttpTest
  Determine hard-linking and nlink support per FileStore
  Support reading and writing cookies.
  Repository: Add getIdentifier() method to avoid instanceof operator
  Update to Orbit R20190602212107
  PacketLineIn: Deprecate the END constant
  PacketLineIn: Add an iterator over strings in the input stream
  Replace most usages of PacketLineIn.END with PacketLineIn.end()
  PacketLineIn: Deprecate DELIM constant
  Replace trivial reference comparison of PacketLineIn.{DELIM,END}
  PacketLineIn: Rename isDelim to isDelimiter
  ProtocolV2ParserTest: Fix typo in comment
  Upgrade Bouncy Castle to 1.61
  Update to Orbit R20190531194818 and rollback update to Ant 1.10.6
  cli: Add the --always option to describe
  DescribeCommand: Support the "always" option
  cli: Add the --tags option to describe
  DescribeCommand: Consistenly omit the default value
  Remove excess blank line in FileUtilsTest
  PacketLineIn: Add helper methods to check for END and DELIM
  UploadPackTest: Rename variable to avoid hiding class member
  UploadPackTest: Add missing <> operator on instantiation of ArrayList
  BitmapCalculator: javadoc fixes
  RevWalkUtils: add progress callback to findBranchesReachableFrom
  Upgrade maven-source-plugin to 3.1.0
  Upgrade maven-jar-plugin to 3.1.2
  Upgrade jacoco-maven-plugin to 0.8.4
  BitmapCalculator and its test: add missing license header
  RevWalk: new method createReachabilityChecker()

Change-Id: I4d76c7c0dbe6411c842f3468b709f7df51789c08
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-11 23:30:26 +02:00
Thomas Wolf 1cfcde4853 Handle missing "ours" stage in WorkingTreeIterator.hasCrLfInIndex()
In a delete-modify conflict with the deletion as "ours" there may be
no stage 2 in the index. Add appropriate null checks. Add a new test
for this case, and verify that the file gets added with a single LF
after conflict resolution with core.autocrlf=true. This matches the
behavior of canonical git for this case.

Bug: 547724
Change-Id: I1bafdb83d9b78bf85294c78325e818e72fae53bc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-06-11 14:42:46 +02:00
David Pursehouse 8bd0ba0f15 ErrorProne: Increase severity of FutureReturnValueIgnored to ERROR
The only remaining code where the return value is ignored is in tests.
Update them to store the value and perform a basic assertion.

Change-Id: I29ef5bd5dd0648aac3490f9e47ecc74544109652
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-11 10:51:30 +09:00
Matthias Sohn d8429d27f7 Increase bazel timeout for long running tests
EolRepositoryTest and GcCommitSelectionTest timed out frequently when
running unit tests using bazel with the default timeout "moderate"
(300s). Increase timeout of these tests to "long" (900s).

Change-Id: I43588cf950f55b50f868d9fe9c66d22bd428a54c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-10 23:40:03 +02:00
Thomas Wolf 8a2e221096 Make pull --rebase on an unborn branch do a checkout
A merging pull on an unborn branch was already supported. But a
rebasing pull failed. If the user has pull.rebase = true in his
user config, the pull would try to rebase. Rebasing needs a parent
commit, though. Native git handles this case:

  git init
  git remote add origin <URI>
  git pull --rebase origin master

Check up front in PullCommand for the unborn head and just do a
checkout in this case. MergeCommand already has similar code.

Bug: 544965
Change-Id: I1277e1ac0b0364b4623fd791f3d6b07bd5f58fca
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-06-10 16:33:39 +02:00
Thomas Wolf f24ad3da66 Handle escaped CR-LF in git config files
Canonical git treats CR-LF in config files as LF.[1][2] JGit does so,
too, except when escaped as a line continuation. Correct this and
treat the sequence \-CR-LF as a line continuation.

[1] https://github.com/git/git/commit/db2c075d9
[2] https://github.com/git/git/blob/v2.21.0/config.c#L485

Bug: 545850
Change-Id: I51e7378a22c21b3baa3701163c423d04c900af5a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-08 01:43:22 +02:00
Thomas Wolf 10ab407fa6 DescribeCommand: use glob match instead of path match
Otherwise tags may fail to match if their name contains slashes.
Canonical git also uses its wildcard matcher in glob mode.[1]

[1] https://github.com/git/git/blob/v2.21.0/builtin/describe.c#L182

Bug: 546703
Change-Id: I122c7959974fa1fc6a53dfc65837e4314a8badd4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-06-07 17:19:07 +02:00
Thomas Wolf d2600693bd Fix off-by-one error in RebaseTodoFile when reading a todo file
Commit messages of length 1 were not read. 'lineEnd' is the offset
of the last character in the line before the terminating LF or CR-LF,
and 'nextSpace' is actually the offset of the character _after_ the
next space. With a one-character commit message, nextSpace == lineEnd.

The code also assumes the commit message to be optional, but actually
failed in that case because it read beyond the line ending. Fix that,
too.

Add a test case for reading a todo file.

Bug: 546245
Change-Id: I368d63615930ea2398a6230e756442fd88870654
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-06-07 16:19:31 +02:00
David Pursehouse 00f840dcd1 Consistently use "!isEmpty()" to detect non-empty list
Replace "size() > 0" with "!isEmpty()" where appropriate.

In the Status implementation we can drop the check; the subsequent
loop will only execute when the list is non-empty anyway.

Change-Id: I355aff551a603373e702a9d44304f087b476263c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-07 12:12:44 +09:00
David Pursehouse 6518d63317 NetscapeCookieFileTest: Split HttpCookiesMatcher to own class
The bazel build fails due to NetscapeCookieFileTest's internal class not
being visible to TransportHttpTest.

Split the file out to its own class in the util package, so it's visible
to both.

Change-Id: I69236026eecb9d08a9a66e51752a80ea522b0c6a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-07 10:51:21 +09:00
David Pursehouse 9f69656748 Bazel: Add missing dependency on mockito for TransportHttpTest
Change-Id: I54eff21c7aa8ee4b3b4a2fea60c5bca359256808
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-07 10:22:13 +09:00
Konrad Windszus d7bd2e700c Support reading and writing cookies.
The git config entries "http.cookieFile" and
"http.saveCookies" are correctly evaluated.

Bug: 488572
Change-Id: Icfeeea95e1a5bac3fa4438849d4ac2306d7d5562
Signed-off-by: Konrad Windszus <konrad_w@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-06 19:02:17 +02:00
Matthias Sohn d9c569b747 Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  Prepare 5.3.3-SNAPSHOT builds
  JGit v5.3.2.201906051522-r
  Prepare 5.1.9-SNAPSHOT builds
  JGit v5.1.8.201906050907-r
  Test detecting modified packfiles
  Enhance fsTick() to use filesystem timer resolution
  Add debug trace to measure time needed to open pack index
  Extend FileSnapshot for packfiles to also use checksum to detect changes
  Wait opening new packfile until it can't be racy anymore
  Avoid null PackConfig in GC
  Add FileSnapshot test testing recognition of file size changes
  Capture reason for result of FileSnapshot#isModified
  Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
  Tune max heap size for tests
  Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
  ObjectDirectory: fix closing of obsolete packs
  Update API filters for methods added to fix bugs
  Bazel: Increase severity of most error-prone checks to ERROR
  Enable error-prone checks by default
  Add bazel options to align with gerrit's
  Include filekey file attribute when comparing FileSnapshots
  Measure file timestamp resolution used in FileSnapshot
  Fix FileSnapshot's consideration of file size
  Fix API problem filters
  Fix API problem filters

Change-Id: Ie1c7e4752661aec9e1754660934921224e2408eb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-06 17:14:31 +02:00
Matthias Sohn fcfe1299c3 Prepare 5.3.3-SNAPSHOT builds
Change-Id: Idebf272835cb22c21257c88b414ada802629b665
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-05 23:35:14 +02:00
Matthias Sohn d6c7c42a1b JGit v5.3.2.201906051522-r
Change-Id: I41b4926f4bfa896ef2a13c35e93e08071ae949a3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-05 21:23:13 +02:00