Commit Graph

2504 Commits

Author SHA1 Message Date
Matthias Sohn 4018709eb9 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Prepare 5.1.9-SNAPSHOT builds
  JGit v5.1.8.201906050907-r
  Test detecting modified packfiles
  Enhance fsTick() to use filesystem timer resolution
  Add debug trace to measure time needed to open pack index
  Extend FileSnapshot for packfiles to also use checksum to detect changes
  Wait opening new packfile until it can't be racy anymore
  Avoid null PackConfig in GC
  Add FileSnapshot test testing recognition of file size changes
  Capture reason for result of FileSnapshot#isModified
  Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
  Tune max heap size for tests
  Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
  ObjectDirectory: fix closing of obsolete packs
  Include filekey file attribute when comparing FileSnapshots
  Measure file timestamp resolution used in FileSnapshot
  Fix FileSnapshot's consideration of file size
  Fix API problem filters

Change-Id: I3ac77bfa03f7436de12ab86e1bba29afee5ccd01
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-05 20:58:16 +02:00
Matthias Sohn 57ccca75e6 Prepare 5.1.9-SNAPSHOT builds
Change-Id: I60571e4e0bea04bb2c25ef3d0332a9ab6895db06
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-05 15:23:56 +02:00
Matthias Sohn 2fe587fcfd JGit v5.1.8.201906050907-r
Change-Id: Iae0ffe161df2ca8a800d21688d6b7d7419dfb640
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-05 15:06:56 +02:00
Christian Halstrick 5f8d91fded Test detecting modified packfiles
Test that JGit detects that packfiles have changed even if they are
repacked multiple times in one tick of the filesystem timer.

Test that this detection works also when repacking doesn't change the
length or the filekey of the packfile. In this case where a modified
file can't be detected by looking at file metadata JGit should still
detect too fast modification by racy git checks and trigger rescanning
the pack list and consequently rereading of packfile content.

Change-Id: I67682cfb807c58afc6de9375224ff7489d6618fb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-05 14:57:02 +02:00
Matthias Sohn 9a7d3b053a Extend FileSnapshot for packfiles to also use checksum to detect changes
If the attributes of FileSnapshot don't detect modification of a
packfile read the packfile's checksum and compare it against the
checksum cached in the loaded packfile.

Since reading the checksum needs less IO than reloading the complete
packfile this may help to reduce the overhead to detect modficiation
when a gc completes while ObjectDirectory scans for packfiles in another
thread.

Bug: 546891
Change-Id: I9811b497eb11b8a85ae689081dc5d949ca8c4be5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-04 18:15:17 +02:00
David Pursehouse e0133b9440 Replace most usages of PacketLineIn.END with PacketLineIn.end()
PacketLineIn.END is only referenced in tests. Replace most of those
with a new package visible end() method.

Remaining usages of PacketLineIn.END are in the form:

  while ((line = pckIn.readString()) != PacketLineIn.END) {

and are not trivial replacements, hence are not touched in this change.

Change-Id: Id77c5321ddcad127130b246bde8f08736e60e1ea
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-03 08:07:24 +09:00
David Pursehouse e99aaa5f79 PacketLineIn: Deprecate DELIM constant
Deprecate DELIM with the intention of making it private in a future
release.

Callers that want to test if a packet line string is the delimiter
should use the isDelimiter(String) method.

The only other references to DELIM in the JGit code are in tests. For
those, introduce a package visible delimiter() method.

Change-Id: I21e8bbac0ffb9ef710c9753e23435416b09a4891
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-03 08:07:24 +09:00
David Pursehouse 7377b8dd01 Replace trivial reference comparison of PacketLineIn.{DELIM,END}
Replace reference comparisons of PacketLineIn's DELIM and END strings
with usage of the helper methods isDelimiter() and isEnd().

Change-Id: I52dcfc4ee9097f1bd6970601c716701847d9eebd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-02 18:32:59 +09:00
David Pursehouse 07a07a75d6 PacketLineIn: Rename isDelim to isDelimiter
Change-Id: I279808b8ddd8a9230a35582c00867b811633dfe8
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-02 18:32:55 +09:00
David Pursehouse 5862a5e222 ProtocolV2ParserTest: Fix typo in comment
Change-Id: I22b07179ba6e2517ae3e178c31fde20987cead34
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-06-02 11:18:55 +09:00
Brandon Weeks f4443ef3f9 Upgrade Bouncy Castle to 1.61
Also now use JcaKeyBoxBuilder constructor in
BouncyCastleGpgKeyLocator.readKeyBoxFile(Path).

CQ: 19868
CQ: 19869
CQ: 19870
Change-Id: I45bd80e158aecd90448b0c7e59615db27aaef892
Signed-off-by: Brandon Weeks <bweeks@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-06-01 01:39:26 +02:00
Sebastian Schuberth 237c76fddf DescribeCommand: Support the "always" option
See:

https://git-scm.com/docs/git-describe#Documentation/git-describe.txt---always

Extend the tests accordingly.

Change-Id: Ibfcda338a246c8cba0df6b6e7b9bad76c9f8b593
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2019-05-31 17:11:15 +02:00
Matthias Sohn 22352e4751 Remove excess blank line in FileUtilsTest
Change-Id: I0d617feb7baacf9917b1da605fcb75f7d4ccd184
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-31 09:40:59 +09:00
David Pursehouse d2425e16f4 PacketLineIn: Add helper methods to check for END and DELIM
These methods will allow clients to check for END and DELIM without
doing a reference comparison on the String objects, which raises
warnings from Error Prone.

Change-Id: I9e7e59843553ed4488ee8e864033198bbb60d67c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-30 20:31:39 +09:00
David Pursehouse 9f5d271b12 UploadPackTest: Rename variable to avoid hiding class member
Change-Id: I9e02642b2f56a735681e58b8c8d6e3545ef9093b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-30 20:18:28 +09:00
David Pursehouse cd1d2d6fe3 UploadPackTest: Add missing <> operator on instantiation of ArrayList
Change-Id: I348cbcd75252d7857ada843dc496d445ee1d62fb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-05-30 20:17:24 +09:00
Matthias Sohn 5d0286eb7c Add FileSnapshot test testing recognition of file size changes
Change-Id: Ibcd76a5e6e4183ada0be1d4436ce957243f2094d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-29 23:41:45 +02:00
Matthias Sohn 43b06f51f9 Capture reason for result of FileSnapshot#isModified
This allows to verify the expected behavior in
FileSnapshotTest#testSimulatePackfileReplacement and enables extending
FileSnapshot for packfiles to read the packfile's checksum as another
criterion to detect modifications without reading the full content.

Also add another field capturing the result of the last check if
lastModified was racily clean.

Remove unnecessary determination of raciness in the constructor. It was
determined twice in all relevant cases.

Change-Id: I100a2f49d7949693d7b72daa89437e166f1dc107
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-29 23:41:45 +02:00
Matthias Sohn d31a0c0b5d Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
NTFS does not support FileKey hence ignore this test on Windows.

Change-Id: I7b53a591daa5e03eb5e401b5b26d612ab68ce10d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-29 23:41:43 +02:00
Matthias Sohn 1e8b68cbc5 Tune max heap size for tests
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].

Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test

Decrease it to 768m for
- org.eclipse.jgit.test

[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console

Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit e19e859977)
2019-05-29 23:39:55 +02:00
Matthias Sohn 0e33b32ab7 Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
testNewFileNoWait() was identical to testNewFileWithWait() but claims it
doesn't wait at all. Hence remove the waits.

Change-Id: I49b8ca5cb49a43c55fe61870c18c42f32fb4b74d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-27 10:49:51 +02:00
Matthias Sohn 73ba6658cf BitmapCalculator and its test: add missing license header
Change-Id: I3c4a8b1e9159b0553aa95213bb82628370b6c036
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-23 02:12:16 +02:00
Matthias Sohn 0988276746 Prepare 5.4.0-SNAPSHOT builds
Change-Id: I9df6fccee253d4087f9afab4cb46e0a40b8a5699
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 22:52:43 +02:00
Matthias Sohn 72998dd5ee JGit v5.4.0.201905221418-m3
Change-Id: I8a7e687acfcf8c341abd726ae60b5ee173422215
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 20:18:14 +02:00
Matthias Sohn e19e859977 Tune max heap size for tests
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].

Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test

Decrease it to 768m for
- org.eclipse.jgit.test

[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console

Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 14:42:43 +02:00
Matthias Sohn 91101414ae Include filekey file attribute when comparing FileSnapshots
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).

Some filesystems expose unique file identifiers, e.g. inodes in Posix
filesystems which are named filekeys in Java's BasicFileAttributes. Use
them as another means to detect file modifications based on stat
information.

Running git gc on a repository yields a new packfile with the same id as
a packfile which existed before the gc if these packfiles contain the
same set of objects. The content of the old and the new packfile might
differ if a different PackConfig was used when writing the packfile.
Considering filekeys in FileSnapshot may help to detect such packfile
modifications.

Bug: 546891
Change-Id: I711a80328c55e1a31171d540880b8e80ec1fe095
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 08:13:56 +02:00
Matthias Sohn b513b77477 Measure file timestamp resolution used in FileSnapshot
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].

Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.

Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.

Measured Java file timestamp resolution: [2]

[1] https://github.com/git/git/blob/master/Documentation/technical/racy-git.txt
[2] https://docs.google.com/spreadsheets/d/1imy0y6WmRqBf0kjCxzxj2X7M50eIVfa7oaUIzEOHmjo

Bug: 546891
Change-Id: I493f3b57b6b306285ffa7d392339d253e5966ab8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-22 08:13:22 +02:00
Jonathan Tan bab685488d Recognize ReachabilityCheckerTestCase as helper
4e196faa1b ("ReachabilityChecker: Default implementation with a
RevWalk", 2019-05-15) added ReachabilityCheckerTestCase but did not add
it as a test helper in the corresponding BUILD file, making tests no
longer runnable with Bazel. Resolve this issue.

Change-Id: Iccc00b0d169dbaa137e130ce2ddd1b7669960b52
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
2019-05-20 17:02:07 -07:00
Jonathan Tan 7121fa6fec Merge changes from topic 'reachability-2'
* changes:
  UploadPack: Use reachability checker to validate non-advertised wants
  BitmappedReachabilityChecker: Reachability check using bitmaps
  BitmapCalculator: Get the reachability bitmap of a commit
  ReachabilityChecker: Default implementation with a RevWalk
2019-05-20 16:04:28 -04:00
Carsten Hammer 84fc5c908d Convert to lambda or member reference
Convert anonymous inner classes to lambda expressions or member
references

Bug: 545856
CQ: 19537
Change-Id: I621431c178e8b99316314602f7c66c9a36f9ae98
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-16 02:58:43 -04:00
Ivan Frade 601b0ae577 BitmappedReachabilityChecker: Reachability check using bitmaps
The "basic" reachability check walks the graph starting from the tips
marking things as "uninteresting". If the target commit is marked as
"uninteresting" it was reached; it is reachable from those tips.

This requires a lot of walking and can be solved directly with bitmaps.
Most of the time the bitmaps are already calculated or a short walk
away.

This should improve the performance of reachability checks, for example
in Gitiles.

Change-Id: I83d33271f58d95d2dc9ed151967b3eda513c99f7
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-05-15 16:36:20 -07:00
Ivan Frade b6da4b34cc BitmapCalculator: Get the reachability bitmap of a commit
To make reachability checks with bitmaps, we need to get the
reachability bitmap of a commit, which is not always precalculated.

There is already a class returning such bitmap (BitmapWalker) but it
does too much unnecessary work: it calculates ALL reachable objects from
a commit (i.e. including trees and blobs), when for reachability the
commits are just enough.

Introduce BitmapCalculator to get the bitmap of a commit: either because
it is precalculated or generating it with a walk only over commits.

Change-Id: Ibb6c78affe9eeaf1fa362a06daf4fd2d91c1caea
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-05-15 15:04:00 -07:00
Ivan Frade 4e196faa1b ReachabilityChecker: Default implementation with a RevWalk
It is common to check if a certain commit is reachable from some
starting points. For example gitiles does it to check if a commit
is visible to a user based on its permissions.

Offer this functionality in JGit.

Split the interface as the next commit will introduce an implementation
using bitmap indices.

Change-Id: I0933b305c8d734f7a64502910ff4d9ef4fc92ae1
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-05-15 14:55:54 -07:00
Thomas Wolf c33d2bfb9f Apache MINA sshd client: test reading encrypted ed25519 keys
Add encrypted ed25519 keys in the tests; sshd 2.2.0 can finally
decrypt encrypted new-style OpenSSH key files. (Needs the "unlimited
strength" JCE, which is the default since Java 8u161. On older JREs,
users should install the policy files available from Oracle.)

The "expensive" key added has been generated with OpenSSH's
ssh-keygen -t ed25519 -a 256, i.e., with 256 bcrypt KDF rounds
instead of the default 16. On my machine it takes about 2sec to
decrypt.

Bug: 541703
Change-Id: Id3872ca2fd75d8f009cbc932eeb6357d3d1f267c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-05-06 15:22:08 +02:00
Thomas Wolf 86cee68e0d Apache MINA sshd client: adapt to sshd 2.2.0
Update target platforms, maven and bazel builds to use sshd 2.2.0.

Adapt internal classes to changed sshd interfaces and remove previous
work-arounds for asking repeatedly for key passwords and for loading
keys lazily; both are now done by sshd.

CQ: 19034
CQ: 19035
Bug: 541425
Change-Id: I85e1df6ebb8a94953a912d9b2b8a7b5bdfbd608a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-05-06 15:22:05 +02:00
Matthias Sohn 3c92025c1b Merge branch 'stable-5.3'
* stable-5.3:
  Prepare 5.3.2-SNAPSHOT builds
  JGit v5.3.1.201904271842-r
  Prepare 5.2.3-SNAPSHOT builds
  JGit v5.2.2.201904231744-r
  Revert 4678f4b and provide another solution for bug 467631
  Apache MINA sshd: make sendKexInit() work also for re-keying
  Prepare 5.1.8-SNAPSHOT builds
  JGit v5.1.7.201904200442-r
  ObjectUploadListener: Add callback interface
  Prepare 4.11.9-SNAPSHOT builds
  JGit v4.11.8.201904181247-r
  Prepare 4.9.11-SNAPSHOT builds
  JGit v4.9.10.201904181027-r
  Prepare 4.7.10-SNAPSHOT builds
  JGit v4.7.9.201904161809-r
  Prepare 4.5.8-SNAPSHOT builds
  JGit v4.5.7.201904151645-r
  Remember the cause for invalidating a packfile
  Fix API problem filters
  Fix pack files scan when filesnapshot isn't modified

Change-Id: I8a8671f7767444a77b809bd66a27d776c8332736
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-29 00:10:43 +02:00
Matthias Sohn 1342942cc8 Prepare 5.3.2-SNAPSHOT builds
Change-Id: Iedd56602acc89783387098c7f92ce0e5bad091e0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-28 00:54:08 +02:00
Matthias Sohn 38049a008d JGit v5.3.1.201904271842-r
Change-Id: If3c323acfd2b6933f7d4fbec480cd4e82224f701
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-28 00:41:59 +02:00
Matthias Sohn 32f0963b84 Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Prepare 5.2.3-SNAPSHOT builds
  JGit v5.2.2.201904231744-r
  Revert 4678f4b and provide another solution for bug 467631
  Apache MINA sshd: make sendKexInit() work also for re-keying
  Prepare 5.1.8-SNAPSHOT builds
  JGit v5.1.7.201904200442-r
  ObjectUploadListener: Add callback interface
  Prepare 4.11.9-SNAPSHOT builds
  JGit v4.11.8.201904181247-r
  Prepare 4.9.11-SNAPSHOT builds
  JGit v4.9.10.201904181027-r
  Prepare 4.7.10-SNAPSHOT builds
  JGit v4.7.9.201904161809-r
  Prepare 4.5.8-SNAPSHOT builds
  JGit v4.5.7.201904151645-r
  Remember the cause for invalidating a packfile
  Fix API problem filters
  Fix pack files scan when filesnapshot isn't modified

Change-Id: Ie7e572ac7e346f21fe0c387d7448be168a9c127a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-28 00:30:47 +02:00
Matthias Sohn a72e0b0188 Prepare 5.2.3-SNAPSHOT builds
Change-Id: I6ae3db901d986467128326073d4ba70406ae8385
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-24 00:04:44 +02:00
Matthias Sohn 1d297c3ddc JGit v5.2.2.201904231744-r
Change-Id: I4ed2aff28bff702a8c1b42814acb04c7ef9025a7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-23 23:43:50 +02:00
Thomas Wolf 8277f62897 Revert 4678f4b and provide another solution for bug 467631
Making gitlinks and folders match in a tree walk was the wrong
approach to fix bug 467631. The problem is that in such a conflict
the tree walk may then not descend into the folder.

Revert the changes to Paths.java and PathsTest.java from commit
4678f4b. Instead test for the problem case from bug 467631 explicitly
in IndexDiff. Add Daniel's test case from bug 545162, and add yet
another test case for DiffEntry.scan() that covers the problem
originally reported in bug 545162.

Bug: 545162
Change-Id: Ie2214c5d5ee32ac6596b621f0f1c7b86d38fa9b7
Also-by: Daniel Veihelmann <daniel.veihelmann@gmail.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-04-20 11:19:34 +02:00
Matthias Sohn c2945bd261 Prepare 5.1.8-SNAPSHOT builds
Change-Id: I331bbaefd42ef94673ae8565c9b2b3af9eadfda0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-20 10:51:17 +02:00
Matthias Sohn 22e41cfd56 JGit v5.1.7.201904200442-r
Change-Id: Ifaa3a88c5e117912a8c691b8b8369dd9a17faebb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-20 10:42:58 +02:00
Matthew DeVore 0a15cb3a2b tree:<depth>: do not revisit tree during packing
If a tree is visited during pack and filtered out with tree:<depth>, we
may need to include it if it is visited again at a lower depth.

Until now we revisit it no matter what the depth is. Now, avoid
visiting it if it has been visited at a lower or equal depth.

Change-Id: I68cc1d08f1999a8336684a05fe16e7ae51898866
Signed-off-by: Matthew DeVore <matvore@gmail.com>
2019-04-16 10:36:59 -07:00
Matthew DeVore 175e66548b tree:<depth> should not traverse overly-deep trees
If we are traversing a tree which is too deep, then there is no need to
traverse the children. Skipping children is much faster than traversing
the possibly thousands of objects which are directly or indirectly
referenced by the tree.

Change-Id: I6d68cc1d35da48e3288b9cc80356a281ab36863d
Signed-off-by: Matthew DeVore <matvore@gmail.com>
2019-04-16 10:36:59 -07:00
Matthew DeVore 93dd2d482a Preliminary support for tree:<depth> filter
This is used when fetching, and in particular to populate a partial
clone or a virtual file system cache as the user navigates. With this,
a client can pre-fetch a few directories deeper than only the current
directory.

depth:0 will omit all trees, and is useful if you only want to fetch
the commits of a repository, or fetch just a single tree or blob object.
depth:1 will fetch only the root tree of all commits fetched. depth:2
will fetch the root tree and all blobs and tree objects directly
referenced from it. depth:3 gets one more level, and so on. depth:#
will not filter a blob or tree that is explicitly marked wanted.

Bitmaps are disabled when this filter is used.

This implementation is quite slow because it iterates over all omitted
objects rather than skipping them. This will be addressed in follow-up
commits.

Change-Id: Ic312fee22d60e32cfcad59da56980e90ae2cae6a
Signed-off-by: Matthew DeVore <matvore@gmail.com>
2019-04-16 10:36:55 -07:00
Carsten Hammer 0b320b2c35 Use Arrays.asList instead of copying array in a for loop
Change-Id: Ie44950f7d2f2f94a0412efb6c274f6e1e31efcd6
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-14 00:19:51 +02:00
Carsten Hammer 6a4c77e619 Use isEmpty() instead of size()==0 where possible
Change-Id: I97f1367a2ea9f1f6146e264c27c3981b842f2a26
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-13 19:53:29 +02:00
Carsten Hammer 6e03645ad8 Use jdk 5 for loop
Replace simple uses of Iterator with a corresponding for-loop.
Also add missing braces on loops as necessary.

Change-Id: I708d82acdf194787e3353699c07244c5ac3de189
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-13 19:33:13 +02:00
Carsten Hammer c0268f899e Join catch sections using multicatch
Change-Id: I1a9112e6a4f938638c599b489cb0858eca27ab91
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-04-13 18:53:58 +02:00
Carsten Hammer e876a70ede Use StringBuilder instead of StringBuffer where possible.
Change-Id: Ifc67c84a3e786cd766f45726733a6d294d2652a1
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-04-11 11:47:40 +09:00
David Pursehouse dff4f6cc74 ConfigTest: Add test for parsing invalid group header
Test that an exception is raised for an invalid group header:

  [group "foo" ]
     foo = bar

i.e. where there is a space between the group subsection name
and the closing ']'.

Change-Id: I8933ae100b77634b0afb66bb8aa43d24c955799e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-04-11 09:09:47 +09:00
Patrick Hiesel 93a0e50ed0 Introduce RefDatabase#getTipsWithSha1 to list refs pointing to object
Add resolveTipSha1, an inverse of exactRef(String ...), to RefDatabase
and provide a default implementation that runs in O(n) time where n is
the number of refs. For RefTable, provide an implementation that runs
in O(log(n)) time.

[ifrade@google.com: with tests in InMemoryRepositoryTest to exercise
 the reftable code path, too]

Change-Id: I2811ccd0339cdc1c74b42cce2ea003f07a2ce9e1
Signed-off-by: Patrick Hiesel <hiesel@google.com>
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-04-08 17:53:51 -07:00
David Pursehouse 9c2ef2d71e UploadPackTest: Stop using deprecated Transport.setFilterBlobLimit(long)
Replace usage with the recommended setFilterSpec(FilterSpec).

Change-Id: Icc528d175f25234eeb2daa6b4c29a67a7a6d1e0a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-04-08 01:13:00 +02:00
Ivan Frade 9d97d2404f DfsRefDatabase: Keep update index when peeling a reference
The new references created in the peeling do not receive the update
index. In other words, the update index of a reference (if set) is lost
in the peeling.

Pass-through the update index to the newly created references.

Tested via InMemoryRepository, which uses DfsReftableDatabase.

Change-Id: I7ff7c737a9c3366fdec296a4d9b2e51d10227957
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-03-29 22:50:26 +01:00
Han-Wen Nienhuys bbff3ff008 Clarify error message for invalid Windows characters
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I20c37d70fc78d745f83b9ce702777535b16b8dfe
2019-03-29 17:34:17 -04:00
Matthew DeVore cc9ca71a16 Put filter spec information in a dedicated object
This increases type-safety and is ground work for support of the
"tree:<depth>" filter.

Change-Id: Id19eacdcdaddb9132064c642f6d554b1060efe9f
Signed-off-by: Matthew DeVore <matvore@gmail.com>
2019-03-22 16:06:22 -07:00
Matthias Sohn 5efcbaf7c3 Merge branch 'stable-5.3'
* stable-5.3:
  Fix GC to delete empty fanout directories after repacking

Change-Id: I080ddb03c6143f0bfa24707a10a4d926676d32b1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-21 16:47:14 +01:00
Matthias Sohn c23ccd29b5 Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Fix GC to delete empty fanout directories after repacking

Change-Id: I29e1da15396daaf0036bcb92cfb567cc243db5a1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-21 16:45:34 +01:00
Matthias Sohn 88fffedd99 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix GC to delete empty fanout directories after repacking

Change-Id: I0cf4d26bdee5ecee43e723c4176efbce777acabc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-21 16:14:12 +01:00
Matthias Sohn 08a5d2771d Merge branch 'stable-5.0' into stable-5.1
* stable-5.0:
  Fix GC to delete empty fanout directories after repacking

Change-Id: I5c0e7d59f137c27e4588f20f4472d3ea450cd59c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-21 16:07:23 +01:00
Matthias Sohn b29bb5cbb0 Merge branch 'stable-4.11' into stable-5.0
* stable-4.11:
  Fix GC to delete empty fanout directories after repacking

Change-Id: Idce894a24e126e0fbe7bc9b6a3c64318f1a8eb75
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-21 15:55:47 +01:00
Matthias Sohn 1a289e3420 Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
  Fix GC to delete empty fanout directories after repacking

Change-Id: I7118b9c668dcbb0f5435cc613e964c557bfebf01
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-21 15:35:26 +01:00
Matthias Sohn f06de6b575 Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
  Fix GC to delete empty fanout directories after repacking

Change-Id: Ibdbfe08eb290286fa738010bad1c604e857885cd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-21 15:07:05 +01:00
Masaya Suzuki 47e37b0fa4 Use RefMap instead of HashMap
HashMap<String, Ref> has a memory overhead for refs. Use RefMap.

Change-Id: I3fb4616135dacf687cc3bc2b473effc66ccef5e6
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2019-03-20 15:52:14 -07:00
Matthias Sohn 534ede2d81 Fix GC to delete empty fanout directories after repacking
The prune method did not delete empty fanout directories when loose
objects moved to a new pack file but only when loose unreferenced
objects were pruned.

Change-Id: Ia068f4914c54d9cf9f40b75e8ea50759402b5000
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-20 14:10:30 +01:00
Sebastian Schuberth 40e74f8a1a DescribeCommandTest: Trivially fix formatting by removing a blank line
Change-Id: Ibf0343f05b2c08f87e4f3cb0d56a11e797bcde53
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2019-03-18 09:51:58 +01:00
Matthias Sohn 4ade2de34f Prepare 5.3.1-SNAPSHOT builds
Change-Id: I8dc5e47bd19809d684579610edf922abca3cb3cc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 15:10:33 +01:00
Matthias Sohn de4a633201 JGit v5.3.0.201903130848-r
Change-Id: I52256a6d07011030698299d9a97dd01ea825fb7f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 13:47:54 +01:00
Matthias Sohn 639d53721c Prepare 5.1.7-SNAPSHOT builds
Change-Id: Ica716b2ad18f8390e75466430ef81c86ea06f004
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 07:49:33 +01:00
Matthias Sohn a31756fbfc JGit v5.1.6.201903130242-r
Change-Id: Iaf5e451f72dc8512afee99a9f3fb91aa348e3cf5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 07:41:27 +01:00
Matthias Sohn 6a30ffd033 Prepare 4.11.8-SNAPSHOT builds
Change-Id: I92789b0c02e83786961b81556bf2faec5d26b0c1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 02:13:36 +01:00
Matthias Sohn a888f80856 JGit v4.11.7.201903122105-r
Change-Id: I4fc62c304b5a8452ca72a3b305e5732a26120000
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 02:04:31 +01:00
Matthias Sohn 2b64345f53 Prepare 4.9.10-SNAPSHOT builds
Change-Id: I2bdbe822ee623c5f15a26d092136d8602cef06ac
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 01:34:49 +01:00
Matthias Sohn c28c3ab9f0 JGit v4.9.9.201903122025-r
Change-Id: I807d24461b3557eee48294bfaa652b2a13d78113
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-13 01:27:02 +01:00
Matthias Sohn 51f2979f13 Merge branch 'stable-5.3'
* stable-5.3:
  Reduce contention on PackFile.idx() function.
  Use SystemReader in JSchConfigSessionFactoryTest
  Avoid NPE in ObjectId.isId()

Change-Id: I1d13f6fb705258ae6d6e5fa5e733bfacd4f3d0e3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:14:14 +01:00
Matthias Sohn d44225d85c Use SystemReader in JSchConfigSessionFactoryTest
This isolates the test from the concrete system it's running on.
SshSessionFactory reads the user also through SystemReader.

Change-Id: I1c796aa1c498fe3967456d8589e6be0a82ab8f44
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-11 10:28:42 +01:00
Matthias Sohn 90107d53eb Prepare 5.4.0-SNAPSHOT builds
Change-Id: I90a4791f63d0eba23da744c720e869f1830b86e7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-07 00:34:14 +01:00
Matthias Sohn 0295ee1164 Prepare 5.3.0-SNAPSHOT builds
Change-Id: I9b934bcfad3d2091f4da163170348b0c8ef5e732
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-06 22:26:03 +01:00
Matthias Sohn aca889f8cc JGit v5.3.0.201903061415-rc1
Change-Id: I85866af97fe1c0c0a0da83a7605484082fa56b52
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-06 20:03:00 +01:00
David Pursehouse 01cf73c7ab Merge branch 'stable-4.11' into stable-5.0
* stable-4.11:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: If75b149e693005dd3fe06b523e6e6784bedf44c1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-06 14:15:36 +09:00
David Pursehouse 3fc9d43339 Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: I9052e318b5d920770f7c7121d36e3c58df9d5f5a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-06 14:14:40 +09:00
David Pursehouse ae5ea80363 Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: I0847251eb010616a705e0b91df4bdebc225fa95d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-06 14:13:48 +09:00
Matthias Sohn e8bd9556c0 Cancel gc if thread was interrupted
see
https://groups.google.com/d/msg/repo-discuss/oDB2rl3doDc/tFEh5Xt0CAAJ

Change-Id: Ia6d4631c64e065d8b9b09e0b45e7a9ea8ac3f41d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit 882fed0d96)
2019-03-06 02:36:38 +01:00
David Pursehouse 52179ee247 Merge branch 'stable-5.2'
* stable-5.2:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository

Change-Id: Ib972641105cef8089791fc65389c4f43e218620a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-05 10:24:34 +09:00
David Pursehouse c2dbdabfb9 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository

Change-Id: I42b72a3bba3b4c9389d91a1a35a8004836567e7c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-05 10:23:38 +09:00
Matthias Sohn 882fed0d96 Cancel gc if thread was interrupted
see
https://groups.google.com/d/msg/repo-discuss/oDB2rl3doDc/tFEh5Xt0CAAJ

Change-Id: Ia6d4631c64e065d8b9b09e0b45e7a9ea8ac3f41d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-03 18:35:04 +01:00
Thomas Wolf 60cf85a4a3 Fix core.autocrlf for non-normalized index
With text=auto or core.autocrlf=true, git does not normalize upon
check-in if the file in the index contains already CR/LFs. The
documentation says: "When text is set to "auto", the path is
marked for automatic end-of-line conversion. If Git decides that
the content is text, its line endings are converted to LF on
checkin. When the file has been committed with CRLF, no conversion
is done."[1]

Implement the last bit as in canonical git: check the blob in the
index for CR/LFs. For very large files, we check only the first 8000
bytes, like RawText.isBinary() and AutoLFInputStream do.

In Auto(CR)LFInputStream, ensure that the buffer is filled as much as
possible for the isBinary() check.

Regarding these content checks, there are a number of inconsistencies:

* Canonical git considers files containing lone CRs as binary.
* RawText checks the first 8000 bytes.
* Auto(CR)LFInputStream checks the first 8096 (not 8192!) bytes.

None of these are changed with this commit. It appears that canonical
git will check the whole blob, not just the first 8k bytes. Also
note: the check for CR/LF text won't work with LFS (neither in JGit
nor in git) since the blob data is not run through the smudge filter.
C.f. [2].

Two tests in AddCommandTest actually tested that normalization was
done even if the file was already committed with CR/LF.These tests
had to be adapted. I find the git documentation unclear about the
case where core.autocrlf=input, but from [3] it looks as if this
non-normalization also applies in this case.

Add new tests in CommitCommandTest testing this for the case where
the index entry is for a merge conflict. In this case, canonical git
uses the "ours" version.[4] Do the same.

[1] https://git-scm.com/docs/gitattributes
[2] https://github.com/git/git/blob/3434569fc/convert.c#L225
[3] https://github.com/git/git/blob/3434569fc/convert.c#L529
[4] https://github.com/git/git/blob/f2b6aa98b/read-cache.c#L3281

Bug: 470643
Change-Id: Ie7310539fbe6c737d78b1dcc29e34735d4616b88
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 16:18:27 +09:00
David Pursehouse 0368906353 CommitBuilder: Deprecate setEncoding(String)
Deprecate the method in favor of setEncoding(Charset).

Update the only caller in the code base that was still using
the deprecated variant.

Change-Id: I6357f2d0c727007013c72e9d5b7c72a3f5f3f2b1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-02-18 00:14:35 +01:00
Minh Thai 2b7fa04dad Fix bug in copyPackBypassCache's skip 'PACK' header logic
Bug caused the pack to be 12 bytes short when cold cache.  Also added
test for copyPackAsIs method.

Change-Id: Idf8fb0e50d1215245d4b032e2e00df4b218c115f
Signed-off-by: Minh Thai <mthai@google.com>
2019-02-13 12:48:05 -08:00
Matthias Sohn 8cad84ccfb Fix GC.deleteEmptyRefsFolders
This method tried to iterate spurious files which may exist in the
.git/refs folder, e.g. on Mac a .DS_Store may have been created there by
inspecting the folder using the finder application. This led to a
NotDirectoryException when deleteEmptyRefsFolders tried to create an
iterator for such a file entry. Skip files contained in the refs folder
to ensure the method only tries to iterate contained folders but not
files.

Change-Id: I5f31e733072a35db1e93908a9c69a8891ae5c206
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-28 02:04:45 -05:00
Thomas Wolf e7b4d108e1 Enable cloning only specific tags
Single-branch-clone should be able to clone a single tag. Enhance
CloneCommand to accept also full refs of tags in setBranchesToClone().
Make sure we also include fetch ref specs for the fetch command for
tags. This mimics the behavior of native git's single-branch clone:
git clone --branch <tag> --single-branch <URI>

Bug: 542611
Change-Id: I285cf043751d9b0ba71258ee8214c0e5d1191428
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-01-27 22:56:06 +01:00
Thomas Wolf 447e107069 RenameBranchCommand: more consistent handling of short ref names
Several problems:

* The command didn't specify whether it expected short or full names.
* For the new name, it expected a short name, but then got confused
  if tags or both local and remote branches with the same name existed.
* For the old name, it accepted either a short or a full name, but
  again got confused if a short name was given and a tag with the
  same name existed.

With such an interface, one cannot use Repository.findRef() to
reliably find the branch to rename. Use exactRef() for the new
name as by the time the Ref is needed its full name is known.
For determining the old Ref from the name, do the resolution
explicitly: first try exactRef (assuming the old name is a full
name); if that doesn't find anything, try "refs/heads/<old>" and
"refs/remotes/<old>" explicitly. Throw an exception if the name
is ambiguous, or if exactRef returned something that is not a
branch (refs/tags/... or also refs/notes/...).

Document in the javadoc what kind of names are valid, and add tests.

A user can still shoot himself in the foot if he chooses exceptionally
stupid branch names. For instance, it is still possible to rename a
branch to "refs/heads/foo" (full name "refs/heads/refs/heads/foo"),
but it cannot be renamed further using the new short name if a branch
with the full name "refs/heads/foo" exists. Similar edge cases exist
for other dumb branch names, like a branch with the short name
"refs/tags/foo". Renaming using the full name is always possible.

Bug: 542446
Change-Id: I34ac91c80c0a00c79a384d16ce1e727c550d54e9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-01-23 19:10:36 +01:00
Gunnar Wagenknecht c4420d2434 Allow to check for signing key
The new API is intended for UIs to check if signing will be possible or
would fail

Bug: 543579
Change-Id: I6ce1fd4210e46d49dcdf420c99d08c93e022136c
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
2019-01-22 03:38:36 -05:00
Thomas Wolf 55966dc592 Handle premature EOF in BundleFetchConnection
BundleFetchConnection.readLine() must abort on EOF, otherwise
it gets stuck in an endless loop.

Bug: 543390
Change-Id: I4cb3428560277888af114b928950d620bb6564f9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-01-22 01:49:03 -05:00
David Pursehouse 95d74f22fb TemporaryBufferTest: Open TemporaryBuffer in try-with-resource
Except the cases where the warning was suppressed in bf757cd because
they are explicitly testing functionality that is not possible in t-w-r.

Change-Id: I6c66eb31a2314028e09a9213dc695cd163f907e9
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-21 13:59:21 +09:00
David Pursehouse 6503a67e46 SubmoduleWalkTest: Open Repository in try-with-resource
Change-Id: I9c4ee4e4c55f48db1f21e18b212657e4dd3de053
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-21 13:40:48 +09:00
David Pursehouse df774825f5 SideBandOutputStreamTest: Open SideBandOutputStream in try-with-resource
Change-Id: I34041a556a5a83afcd0c1bab00e5d8088c30ea3f
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-21 13:36:31 +09:00
David Pursehouse ecd16a7906 UnionInputStreamTest: Open UnionInputStream in try-with-resource
The tests were written for Java 7 which did not have AutoCloseable
and the try-with-resource concept. When the project was updated to
build with Java 8, the warnings were suppressed.

Remove the suppressions and convert to use try-with-resource.

Change-Id: Ic805bd571c4a2e4376ce5e7c34ca7ac86cbf5104
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-21 13:28:22 +09:00