Commit Graph

5503 Commits

Author SHA1 Message Date
Yunjie Li e250482c7a PackBitmapIndex: Remove convertedBitmaps in the Remapper
The convertedBitmaps serves for time-optimization purpose. But it's
actually not saving time much but using lots of memory. So remove the
field here to save memory.

Currently the remapper class is only used in the construction of the
bitmap index file. And during the preparation of the file, we're only
getting bitmaps from the remapper when finding objects accessible from
a commit, so bitmap associated with each commit will only be fetched once
and thus the convertedBitmaps would hardly be read, which means that it's
not saving time.

Change-Id: Ic942a8e485135fb177ec21d09282d08ca6646fdb
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li dcb0265436 PackBitmapIndex: Reduce memory usage in GC
Currently, the garbage collection is consistently failing for some large
repositories in the building bitmap phase, e.g.Linux-MSM project:
https://source.codeaurora.org/quic/la/kernel/msm-3.18

Historically, bitmap index creation happened in 3 phases:
1. Select the commits to which bitmaps should be attached.
2. Create all bitmaps for these commits, stored in uncompressed format
in the PackBitmapIndexBuilder.
3. Deltify the bitmaps and write them to disk.

We investigated the process. For phase 2 it's most efficient to create
bitmaps starting with oldest commit and moving to the newest commit,
because the newer commits are able to reuse the work for the old ones.
But for bitmap deltification in phase 3, it's better when a newer
commit's bitmap is the base, and the current disk format writes bitmaps
out for the newest commits first.

This change introduces a new collection to hold the deltified and
compressed representations of the bitmaps, keeping a smaller subset of
commits in the PackBitmapIndexBuilder to help make the bitmap index
creation more memory efficient.

And in this commit, we're setting DISTANCE_THRESHOLD to 0 in the
PackWriterBitmapPreparer, which means the garbage collection will not
have much behavoir change and will still use as much memory as before.

Change-Id: I6ec2c3e8dde11805af47874d67d33cf1ef83660e
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li 067d946090 PackBitmapIndex: Add AddToBitmapWithCacheFilter class
Add a new revwalk filter, AddToBitmapWithCachedFilter. This filter updates
a client-provided {@code BitmapBuilder} as a side effect of a revwalk.
Similar to {@code AddToBitmapFilter}, it short circuits the walk when it
encounters a commit which is included in the provided bitmap's BitmapIndex.
It also short circuits the walk if it encounters the client-provided
cached commit.

Change-Id: I62cb503016f4d3995d648d92b82baab7f93549a9
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li b1d4b45708 PackBitmapIndex: Add util methods and builder to BitmapCommit
Add some utility methods and a builder class for BitmapCommit class in
preparation for improving the memory footprint of GC's bitmap generation
phase.

Change-Id: Ice3d257fc26f3917a65a64eaf53b508b89043caa
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li d23254ee57 PackBitmapIndex: Move BitmapCommit to a top-level class
Move BitmapCommit from inside the PackWriterBitmapPreparer to a new
top-level class in preparation for improving the memory footprint of GC's
bitmap generation phase.

Change-Id: I4d404a5b3a34998b441d23105197f33d32d39670
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li 840e414d0b Refactor: Make retriveCompressed an method of the Bitmap class
Make retrieveCompressed() a method of Bitmap interface to avoid type
casting and later reuse in improving the memory footprint of GC's bitmap
generation phase.

Change-Id: I098d85105cf17af845d43b8c71b4ca48b02fd7da
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:05 -07:00
Matthias Sohn a379d007db Fix downloading LFS Object fails behind proxy
When downloading LFS objects also accept response code 203 as successful
download. This response may be seen when downloading via a proxy.

Bug: 563022
Change-Id: Iee85fdb451b33369d08859872e5bfc2a67dffa6d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-12 10:26:45 -04:00
Terry Parker 0642e49f97 Merge changes I6543c2e1,I21ed029d
* changes:
  ReceivePack: adding IterativeConnectivityChecker
  Moving transport/internal -> internal/transport
2020-05-11 17:40:20 -04:00
Demetr Starshov 9075beefb1 ReceivePack: adding IterativeConnectivityChecker
Introduce an IterativeConnectivityChecker which runs a connectivity
check with a filtered set of references, and falls back to using the
full set of advertised references.

It uses references during first check attempt:
- References that are ancestors of an incoming commits (e.g., pushing
a commit onto an existing branch or pushing a new branch based on
another branch)
- Additional list of references we know client can be interested in
(e.g. list of open changes for Gerrit)

We tested it inside Google and it improves connectivity for certain
topologies. For example connectivity counts for
chromium.googlesource.com/chromium/src:

percentile_50: 1923 (was: 22777)
percentile_90: 23272 (was: 353003)
percentile_99: 345522 (was: 353435)

This saved ~2 seconds on every push to this repository.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I6543c2e10ed04622ca795b195665133e690d3b10
2020-05-08 17:57:20 -07:00
Demetr Starshov 519cb1e91b Moving transport/internal -> internal/transport
Moving transport related internal classes into dedicated subpackage in
o/e/j/internal package.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I21ed029d359f5f7d8298f102efbb4b1dcdf404ad
2020-05-08 17:57:20 -07:00
Nail Samatov f30e48914e Fix error occurring during checkout
Fix NullPointerException occurring when calling
CheckoutCommand with forced == true option when
the branch isn't changed and there is deleted
uncommitted file.

Change-Id: I99bf1fc25e6889f07092320d7bc2772ec5d341b5
Signed-off-by: Nail Samatov <sanail@yandex.ru>
2020-05-07 12:18:44 +03:00
Matthias Sohn 231c44d553 ApplyCommand: use Files#copy to copy file
This should be faster.

Change-Id: I404ec5e66731b3cf7a8e621cf1ff8748d109ea69
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-06 10:16:35 +02:00
Jack Wickham b6d70a66ae Apply hunks when renaming or copying from patch files
When applying a patch that contains renames or copies using ApplyCommand,
also apply all hunks that apply to the renamed or copied file.

Change-Id: I9f3fa4370458bd7c14beeb2e2b49e846d70203cb
Signed-off-by: Jack Wickham <jwickham@palantir.com>
2020-05-06 08:50:08 +02:00
Jack Wickham d69c0ef5bd Create parent directories when renaming a file in ApplyCommand
Before this change, applying a patch will fail if the destination directory
doesn't exist; after, the necessary parent directories are created.

If renaming the file fails, the directories won't be deleted, so this change
isn't atomic. However, ApplyCommand is already not atomic - if one hunk fails
to apply, other hunks still get applied - so I don't think that is a blocker.

Change-Id: Iea36138b806d4e7012176615bcc673756a82f365
Signed-off-by: Jack Wickham <jwickham@palantir.com>
2020-05-06 08:50:08 +02:00
Ivan Frade 0a2a094fea Reduce BitmappedObjectReachabilityChecker visibility
ObjectReachabilityChecker interface is the only public API. The
implementation is instantiated by ObjectWalk and doesn't need to be
visible outside the package.

Change-Id: I5b97bb98990cded637686bdc15c9655330b7780f
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-05-04 11:45:15 -07:00
Thomas Wolf 9f07528ff8 Add missing @since to new API
Change-Id: Ie7c3481aba515d9c9b0a152db17b5a9dc74e7ede
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-04 09:30:24 +02:00
Ivan Frade 70ec9a83b8 ObjectReachabilityCheckers: Make walk member final
It is only assigned on initialization.

Change-Id: I36f68b3f511236e66cc7a4f56ba49252b317276a
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-30 10:23:29 -07:00
Ivan Frade c9d7285e80 UploadPack: Use more relevant refs first in object reachability check
The bitmap-bassed object reachability checker, tries to find the objects
in the first starter, then adding the second starter... and so on. This
rewards passing the most popular refs first.

Order the refs with heads first, then tags, then others (e.g. changes)
for the object reachability checker. Using streams, delay also the
resolution of the ref to RevObject until necessary.

Change-Id: I9414b76754d7c0ffee1e2eeed6939895c8e92cbe
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade a661e2e9eb UploadPack: Refactor to generalize the object reachability checks
ObjectWalk#createObjectReachabilityChecker() returns the best
implementation for the repo. UploadPack can use the interface and fold
the with/without commits cases in one code path.

Change-Id: I857c11735d1d8e36c3ed8185ff11de8a62e86540
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade 6bc04bdc02 UploadPack: Use BitmappedReachabilityChecker for not advertised wants
Change-Id: Ifea971d5c0309e28a909441ee8a6f1e62397d6d3
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade 003002c1cb revwalk: Introduce bitmap-based object reachability checker
Change-Id: I0b1a2bd21f98894862aab339f8c2e4a417897b89
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:33 -07:00
Ivan Frade 20bb312421 revwalk: Extract ObjectReachabilityChecker interface
Extract ObjectReachabilityChecker interface from the walk-based
implementation, to add a bitmapped based implementation later.

Refactor the test case to use it for both implementations.

Change-Id: Iaac7c6b037723811956ac22625f27d3b4d742139
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 16:56:32 -07:00
Ivan Frade ae26fa19b7 UploadPack: Extract walk-based reachability check
Preparing the code to optimize the bitmap-based object reachability
checker.  We are mirroring first the commit reachability checker
structure (interface + 2 implementations).

Move the walk-base reachability checker to its own class.

This class is public at the moment. Later ObjectWalk will return an
interface and this implementation will be package-private.

Change-Id: Ifac70094e1af137291c3607d95e689992f814b26
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 16:40:52 -07:00
Matthias Sohn c845db3ec2 RefTreeBatch: fix unclosed resource warning
Change-Id: I7d630d11d3ef2ff2b74ec7a7018760b7b4931ad2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:48:14 +02:00
Matthias Sohn 7ae3507998 CherryPickCommand: fix unclosed resource warning
Change-Id: I2dc76efd16d887048a5cb4af2b2d8e6faa4f22cd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:48:14 +02:00
Matthias Sohn 560ef875e3 URIish: suppress non-localized message warning
Change-Id: I3ec37c67ba6f00ad8bf396aa3261dd90f35789ea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:21:06 +09:00
Minh Thai d9f84b0b7c UploadPack: Clear advertised ref map after negotiation
After negotiation phase of a fetch, the advertised ref map is no longer used and
can be safely cleared. For >1GiB repos object selection and packfile writing may
take 10s of minutes. For the chromium.googlesource.com/chromium/src repo, this
advertised ref map is >400MiB. Returning this memory to the Java heap is a major
scalability win.

Change-Id: I00d453c5ef47630c21f199e333e1cfcf47b7e92a
Signed-off-by: Minh Thai <mthai@google.com>
2020-04-23 12:14:02 -07:00
Konrad Windszus 54a2d48008 Fix human name for local .bundle files
Bug: 560903
Change-Id: I15d45330398cc573940265d16a2db29ddce085aa
Signed-off-by: Konrad Windszus <konrad_w@gmx.de>
2020-04-21 07:46:50 +02:00
Matthias Sohn a0aebbd528 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  Remove double blank from sentence start
  Bump Bazel version to 3.0.0

Change-Id: I26c3a8345020239d1c2ec5c6f70a633b43ddab86
2020-04-18 00:30:15 +02:00
Michael Keppler a79c5b1f10 Remove double blank from sentence start
Multiple whitespaces are not normalized when reading properties files,
therefore leading to unwanted space/indentation in console or UI output.

Change-Id: I1f5224fe359e0cac493e0237872afc75dc8b9fbe
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ebbc3efce7)
2020-04-17 14:36:50 +09:00
Michael Keppler ebbc3efce7 Remove double blank from sentence start
Multiple whitespaces are not normalized when reading properties files,
therefore leading to unwanted space/indentation in console or UI output.

Change-Id: I1f5224fe359e0cac493e0237872afc75dc8b9fbe
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
2020-04-14 09:31:50 +02:00
Terry Parker 9c67f680c8 Merge "ReceivePack: Use error message if set" 2020-04-06 11:20:46 -04:00
Minh Thai 0eeed6ad2b Scan through all merged reftables for max/min update indices
Since reftables might have update index ranges that are overlapped.

Change-Id: I8f8215b99a0a978d4dd0155dbaf33e5e06ea8202
Signed-off-by: Minh Thai <mthai@google.com>
(cherry picked from commit 06748c205c)
2020-04-05 17:26:20 -04:00
Alexander Nittka bc4ed530a5 FileUtils: improve delete (Windows)
Ensure files are writable before trying to delete them.

Bug: 408846
Change-Id: I930a547594bba853c33634ae54bd64d236afade3
Signed-off-by: Alexander Nittka <alex@nittka.de>
2020-04-03 19:30:31 +02:00
Thomas Wolf 9aaa58052b Merge branch 'stable-5.7'
* stable-5.7:
  FS.runInShell(): handle quoted filters and hooksPath containing blanks
  Handle non-normalized index also for executable files
  Prepare 5.7.1-SNAPSHOT builds
  JGit v5.7.0.202003110725-r

Change-Id: I8a8580e44bfa05989d476cf22a029abd4fd407c6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-04-03 10:40:30 +02:00
Thomas Wolf 3ae0b5cfeb Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  FS.runInShell(): handle quoted filters and hooksPath containing blanks
  Handle non-normalized index also for executable files

Change-Id: I240377e87c073ee7a621a88e39fc319c59fa037a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-04-03 10:21:00 +02:00
Thomas Wolf 2640d38f14 FS.runInShell(): handle quoted filters and hooksPath containing blanks
Revert commit 2323d7a. Using $0 in the shell command call results in
the command string being taken literally. That was introduced to fix
a problem with backslashes, but is actually not correct.

First, the problem with backslashes occurred only on Win32/Cygwin,
and has been properly fixed in commit 6f268f8.

Second, this is used only for hooks (which don't have backslashes in
their names) and filter commands from the git config, where the user
is responsible for properly quoting or escaping such that the commands
work.

Third, using $0 actually breaks correctly quoted filter commands
like in the bug report. The shell really takes the command literally,
and then doesn't find the command because of quotes.

So revert this change.

At the same time there's a related problem with hooks. If the path to
the hook contains blanks, runInShell() would also fail to find the
hook. In this case, the command doesn't come from user input but is
just a Java File object with an absolute path containing blanks. (Can
occur if core.hooksPath points to such a path with blanks, or if the
repository has such a path.)

The path to the hook as obtained from the file system must be quoted.

Add a test for a hook path with a blank.

This reverts commit 2323d7a1ef.

Bug: 561666
Change-Id: I4d7df13e6c9b245fe1706e191e4316685a8a9d59
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-04-03 08:18:39 +02:00
Matthias Sohn 26b23b2713 Define constants for pack config option keys
Change-Id: Ifb8227cb62370029d6774f2a22b15d6478c713ca
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-04-03 11:49:25 +09:00
Karsten Thoms bd42c92ab3 Fix javadoc typo
Bug: 499934
Change-Id: I0d6ee56fad4472972f69cc1e7cb2a727e95ab6b4
Signed-off-by: Karsten Thoms <karsten.thoms@karakun.com>
2020-04-02 10:54:43 +02:00
Masaya Suzuki 7ad42d7416 ReceivePack: Use error message if set
ReceiveCommand can have an error message. This is shown only for some
cases even if it's set. This change uses the error message if it's set,
and fallback to the default message if unset.

Change-Id: I8d906e71ad08cf49bcdb28caea8fcc66798c68ff
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2020-03-27 12:23:36 -07:00
Thomas Wolf 24f82b533a Handle non-normalized index also for executable files
Commit 60cf85a4 corrected the handling of check-in for files where
the index version is non-normalized, i.e., contains CR-LF line endings.
However, it did so only for regular files, not executable files.

Bug: 561438
Change-Id: I372cc990c5efeb00315460f36459c0652d5d1e77
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-03-25 09:50:12 +01:00
Minh Thai 06748c205c Scan through all merged reftables for max/min update indices
Since reftables might have update index ranges that are overlapped.

Change-Id: I8f8215b99a0a978d4dd0155dbaf33e5e06ea8202
Signed-off-by: Minh Thai <mthai@google.com>
2020-03-20 11:12:34 -07:00
Ivan Frade 7d3b6308fc ResolveMerger: Ignore merge conflicts if asked so
The recursive merge strategy builds a virtual ancestor merging
recursively the common bases (when more than one) between the
want-to-merge commits. While building this virtual ancestor, content
conflicts are ignored, but current code doesn't do so when a file is
removed.

This was spotted in [1], for example. Merging two commits to build the
virtual ancestor bumped into a conflict (modified in one side, deleted
in the other) that stopped the process.

Follow the "spec" and in case of conflict leave the unmerged content in
the index and working trees.

[1] https://android-review.googlesource.com/c/kernel/common/+/1228962

Change-Id: Ife9c32ae3ac3a87d3660fa1242e07854b65169d5
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-03-19 16:36:21 -07:00
Christian Halstrick d22a8a2510 Merge "tag option for clone command" 2020-03-15 18:11:14 -04:00
Alexander Nittka 769606d059 tag option for clone command
Allow explicitly setting the tag option for the remote configuration
when cloning a repository.

Bug: 561021
Change-Id: Iac43268a2bb231ae7599c3255bf555883d34fa32
Signed-off-by: Alexander Nittka <alex@nittka.de>
2020-03-15 15:34:21 +01:00
Alex Spradlin cf8c84c6d1 RevWalk: fix bad topo flags error message
The error message for an Exception thrown by StartGenerator when given
both the TOPO flag and the TOPO_KEEP_BRANCH_TOGETHER flag mentions a
non-existent flag, TOPO_NON_INTERMIX. The error message was introduced
in commit e498d43.

Replace TOPO_NON_INTERMIX with TOPO_KEEP_BRANCH_TOGETHER in the error
message of an Exception thrown by the StartGenerator when the TOPO flag
is provided together with the TOPO_KEEP_BRANCH_TOGETHER flag.

Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
Change-Id: Id24640dc08e96a196508fe38ce144aa7e035082f
2020-03-12 09:04:36 -07:00
Alex Spradlin e498d43186 RevWalk: new topo sort to not mix lines of history
The topological sort algorithm in TopoSortGenerator for RevWalk may mix
multiple lines of history, producing results that differ from C git's
git-log whose man page states: "Show no parents before all of its
children are shown, and avoid showing commits on multiple lines of
history intermixed." Lines of history are mixed because
TopoSortGenerator merely delays producing a commit until all of its
children have been produced; it does not immediately produce a commit
after its last child has been produced.

Therefore, add a new RevSort option called TOPO_KEEP_BRANCH_TOGETHER
with a new topo sort algorithm in TopoNonIntermixGenerator. In the
Generator, when the last child of a commit has been produced, unpop
that commit so that it will be returned upon the subsequent call to
next(). To avoid producing duplicates, mark commits that have not yet
been produced as TOPO_QUEUED so that when a commit is popped, it is
produced if and only if TOPO_QUEUED is set.

To support nesting with other generators that may produce the same
commit multiple times like DepthGenerator (for example, StartGenerator
does this), do not increment parent inDegree for the same child commit
more than once.

Commit b5e764abd2 modified the existing
TopoSortGenerator to avoid mixing lines of history, but it was reverted
in e40c38ab08 because the new behavior
caused problems for EGit users. This motivated adding a new Generator
for the new behavior.

Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
Change-Id: Icbb24eac98c00e45c175b01e1c8122554f617933
2020-03-11 15:39:38 -07:00
Matthias Sohn 0d78edad53 Prepare 5.7.1-SNAPSHOT builds
Change-Id: I29cd1b9872603810f9c4d42424ac326fedf5569f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-11 13:35:23 +01:00
Matthias Sohn 0713f6e962 JGit v5.7.0.202003110725-r
Change-Id: Ia4a9122eeb4eea0f379f6611984cfe1143a32f8d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-11 13:24:54 +01:00
Thomas Wolf 04e16afb05 TransportHttp: support HTTP response 308 Permanent Redirect
RFC 7538[1] added HTTP response code 308, signifying a permanent
redirect that, contrary to the older 301, does not allow changing
the request method from POST to GET.

[1] https://tools.ietf.org/html/rfc7538

Bug: 560936
Change-Id: Ib65f3a3ed75db51d74d1fe81d4abe6fe92b0ca12
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-03-10 11:34:48 +01:00
Thomas Wolf ac83e471d8 Remove unused API problem filters
Change-Id: I137cddaf0aecdf9f4bd1427fa9c1af1af4f5b546
2020-03-10 11:11:14 +01:00
Matthias Sohn d408e41e4e Prepare 5.7.1-SNAPSHOT builds
Change-Id: I23748c20b69a1cfb47c5cd0d7858182d376b3ce1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-09 23:37:43 +01:00
Matthias Sohn 8fab712f75 Prepare 5.8.0-SNAPSHOT builds
Change-Id: I056b45806a82eae80177932e42e3dc806015351a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-09 22:47:10 +01:00
Matthias Sohn f5793bea53 JGit v5.7.0.202003090808-r
Change-Id: Ia23bffa3952687f6107d496a8932d54b11c12412
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-09 14:09:24 +01:00
Matthias Sohn fb0858e9c9 Merge branch 'stable-5.6'
* stable-5.6:
  Silence API errors introduced by 093fbbd1
  Bump Bazel version to 2.2.0
  Expose FileStoreAttributes.setBackground()
  Update reftable storage repo layout

Change-Id: I237eaaed7991e8bbd56a7624f47bbba985330026
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-05 15:44:42 +01:00
Matthias Sohn 2415571370 Silence API errors introduced by 093fbbd1
Change-Id: I1c9d5a25bd06a1152e953c45b683375cb05aa254
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-05 13:53:10 +01:00
Michael Dardis d7f304045a Add validation to hex decoder
Does not fix any issue but prevents user from shooting themselves in the
foot with improper configuration.

Suggested by Demetr Starshov at https://git.eclipse.org/r/#/c/157681/

Change-Id: I006d65022f0a7d4066970825d00080c59404fdc3
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-04 16:34:01 +01:00
Alex Blewitt 093fbbd11e Expose FileStoreAttributes.setBackground()
The FS.setAsyncFileStoreAttributes() static method calls
FileStoreAttributes.setBackground() as its implementation, but there are
other public attributes on this inner class already and there isn't a
real reason why this needs to be private.

By making it public we allow callers to be able to invoke it directly.
Although it doesn't appear that it would make a difference, by calling a
static method on the FS class, all static fields and the transitive
closure of class dependencies must be loaded and initialised, which can
be non-trivial.

Callers referring to FS.setAsyncFileStoreAttributes() may be replaced
with FS.FileStoreAttributes.setBackground() with no change of behaviour
other than improved performance due to less class loading required.

Bug: 560527
Change-Id: I9538acc90da8d18f53fd60d74eb54496857f93a5
Signed-off-by: Alex Blewitt <alex.blewitt@gmail.com>
2020-03-04 15:59:53 +01:00
Han-Wen Nienhuys 30fb4808f2 Update reftable storage repo layout
The change Ic0b974fa (c217d33, "Documentation/technical/reftable:
improve repo layout") defines a new repository layout, which was
agreed with the git-core mailing list.

It addresses the following problems:

* old git clients will not recognize reftable-based repositories, and
  look at encompassing directories.

* Poorly written tools might write directly into
  .git/refs/heads/BRANCH.

Since we consider JGit reftable as experimental (git-core doesn't
support it yet), we have no backward compatibility. If you created a
repository with reftable between mid-Nov 2019 and now, you can do the
following to convert:

  mv .git/refs .git/reftable/tables.list
  git config core.repositoryformatversion 1
  git config extensions.refStorage reftable

Change-Id: I80df35b9d22a8ab893dcbe9fbd051d924788d6a5
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-04 15:45:01 +01:00
Jonathan Nieder fb45137a7c Merge "ObjectWalk: Add null check before skip tree." 2020-03-02 13:37:59 -05:00
Matthias Sohn 2161c1e5e4 Merge branch 'stable-5.6'
* stable-5.6:
  Cygwin expects forward slashes for commands to be run via sh.exe
  Make Logger instances final
  Move array designators from the variable to the type

Change-Id: I9a5dc570deb478525bf48ef526d8cba5b19418bf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-28 23:53:32 +01:00
Thomas Wolf 6f268f8ceb Cygwin expects forward slashes for commands to be run via sh.exe
FS_Win32_Cygwin replaces backslashes by / as a side-effect of
relativize(). When support for core.hooksPath was added, paths were
relativized in a different place using Path.resolve(), which doesn't
do that transformation. As a result hooks could not be run on Cygwin
in some cases.

Do the transformation in FS_Win32_Cygwin.runInShell(). In all other
places, File or Path objects are used, which give no guarantee about
the file separator (typically the system-dependent default separator),
so doing the transformation earlier still wouldn't guarantee that
sh.exe indeed gets a command string using forward slashes.

Bug: 558577
Change-Id: I3c07eb85f0ac7c5628a2e92f990e5cdb7ecf532f
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-02-28 17:18:23 -05:00
Lars Vogel 512ffc9e7f Using for-each loop in jdt
Running the JDT cleanup action for using a for-each loop on jgit

Change-Id: Ie724d8bbdff786ab0167089e90a9914a8135103c
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
2020-02-28 02:39:12 -05:00
David Pursehouse 14a157dff6 Make Logger instances final
Change-Id: Ibb997952917e47bc31a8cbe3863623bc959a8100
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-27 20:32:29 +09:00
David Pursehouse 5a6b6eee35 Move array designators from the variable to the type
As reported by Sonar Lint:

Array designators should always be located on the type for better code
readability. Otherwise, developers must look both at the type and the
variable name to know whether or not a variable is an array.

Change-Id: If6b41fed3483d0992d402d8680552ab4bef89ffb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-27 20:27:31 +09:00
Jonathan Nieder 6586c171b3 Merge "Revert "RevWalk: stop mixing lines of history in topo sort"" 2020-02-26 17:48:18 -05:00
Yunjie Li 7ba261fb5f ObjectWalk: Add null check before skip tree.
currVisit could be null if a blob is marked as start point in
ObjectWalk. Add null check before skipping current tree.

Change-Id: Ic5d876fe2800f3373d136979be6c27d1bbd38dc1
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-02-26 12:03:58 -08:00
Alex Spradlin e40c38ab08 Revert "RevWalk: stop mixing lines of history in topo sort"
This reverts commit b5e764abd2.

PlotWalk uses the TopoSortGenerator, which is causing problems for EGit users
who rely on the emission of commits being somewhat based on date as in the
previous topo-sort algorithm.

Bug: 560529
Change-Id: I3dbd3598a7aeb960de3fc39352699b4f11a8c226
Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
2020-02-26 14:47:20 -05:00
Lajos Olah 61ec1455c0 Do not fail if known hosts file does not contain valid host key
KnownHosts (implementing HostKeyRepository) in Jsch can return null
which could cause NullPointerException in Stream.of(...)

Change-Id: Iddcf5f34f8c8475a85ca7ae018bbe48d1b3fbbc0
Signed-off-by: Lajos Olah <lajos.olah.jr@gmail.com>
2020-02-26 03:57:59 -05:00
Matthias Sohn 25a6bd4d61 Merge branch 'stable-5.6'
* stable-5.6:
  Revert "Prepend hostname to subsection used to store file timestamp resolution"
  SimilarityRenameDetector: Fix inconsistent indentation
  Use indexOf(char) and lastIndexOf(char) rather than String versions
  Reorder modifiers to follow Java Language Specification
  GitmoduleEntry: Remove redundant import of class from same package
  Remove redundant "static" qualifier from enum declarations

Change-Id: Ibb66bef7e8373f81e3e653c9843d986243446d68
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-22 23:36:08 +01:00
Matthias Sohn abb461533f Revert "Prepend hostname to subsection used to store file timestamp resolution"
This reverts commit e102bbed99.

Resolving the hostname comes with a performance penalty. We no longer
store the timestamp resolution in the global git config which might be
copied around to other machines but in a dedicated jgit config meant for
automatically determined options like timestamp resolution. Hence there
is no strong reason anymore to have a hardware specific identifier in
the subsection name of file timestamp resolution options.

Bug: 560414
Change-Id: If8dcabe981eb1792db84643850faa6033f14b1cf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-22 22:57:12 +01:00
Michael Dardis 67b9effc65 Remove use of org.bouncycastle.util.encoders.Hex
Change-Id: I5c1ed0397ef99eb5d4f120da331b66c2d0f1707a
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-22 00:12:20 +01:00
Michael Dardis 4055522359 Remove use of org.bouncycastle.util.io.TeeOutputStream
Bug: 559106
Change-Id: Ife06f21a00d258780a2030745738194098c631cf
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-21 23:45:43 +01:00
Thomas Wolf f2b67be62a Make the IMatcher public API
Provide a static factory method to create a PathMatcher.

Bug: 559526
Change-Id: Ib7a4a1bcc658ac2f2a09d365b5b891669dfd7570
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-02-20 10:40:33 +01:00
David Pursehouse fdabbe67e2 SimilarityRenameDetector: Fix inconsistent indentation
Replace space indentation with tab indentation

Change-Id: Ic130d3bde5d3a73d8f5c6225974153573722d05b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 20:34:15 +09:00
David Pursehouse 6a72f2943d Use indexOf(char) and lastIndexOf(char) rather than String versions
An indexOf or lastIndexOf call with a single letter String can be
made more performant by switching to a call with a char argument.

Found with SonarLint.

As a side-effect of this change, we no longer need to suppress the
NON-NLS warnings.

Change-Id: Id44cb996bb74ed30edd560aa91fd8525aafdc8dd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:24:01 +09:00
David Pursehouse 064834d350 Reorder modifiers to follow Java Language Specification
The Java Language Specification recommends listing modifiers in
the following order:

1. Annotations
2. public
3. protected
4. private
5. abstract
6. static
7. final
8. transient
9. volatile
10. synchronized
11. native
12. strictfp

Not following this convention has no technical impact, but will reduce
the code's readability because most developers are used to the standard
order.

This was detected using SonarLint.

Change-Id: I9cddecb4f4234dae1021b677e915be23d349a380
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:15:43 +09:00
David Pursehouse 2d1acae6d8 GitmoduleEntry: Remove redundant import of class from same package
Change-Id: I613489a937dd53780b0abfd6b1bd0ed964a5ddd8
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:06:18 +09:00
David Pursehouse 9be93b7991 Remove redundant "static" qualifier from enum declarations
Nested enum types are implicitly static.

Change-Id: Id3d7886087494fb67bc0d080b4a3491fb4baac19
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:04:43 +09:00
Matthias Sohn 9e4781878b Merge branch 'stable-5.7'
* stable-5.7:
  Update API problem filter
  Prepare 5.6.2-SNAPSHOT builds
  JGit v5.6.1.202002131546-r
  Simplify ReftableCompactor

Change-Id: I1aa650e030fbaa7a1c898eef1836de529e0c30db
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-19 01:45:18 +01:00
Matthias Sohn 7a940542ee Merge branch 'stable-5.6' into stable-5.7
# By Matthias Sohn (3) and Han-Wen Nienhuys (1)
* stable-5.6:
  Update API problem filter
  Prepare 5.6.2-SNAPSHOT builds
  JGit v5.6.1.202002131546-r
  Simplify ReftableCompactor

Change-Id: I16ed174f9fc662934c3ebaea85a60690efbed1c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-19 01:44:18 +01:00
Alex Spradlin b5e764abd2 RevWalk: stop mixing lines of history in topo sort
The topological sort algorithm in TopoSortGenerator for RevWalk may mix
multiple lines of history, producing results that differ from C git's
git log whose man page states: "Show no parents before all of its
children are shown, and avoid showing commits on multiple lines of
history intermixed." Lines of history are mixed because
TopoSortGenerator merely delays a commit until all of its children have
been produced; it does not immediately produce a commit after its last
child has been produced.

Therefore, when the last child of a commit has been produced, unpop the
commit so that it will be returned upon the subsequent call to next() in
TopoSortGenerator. To avoid producing duplicates, mark commits that
have not yet been produced as TOPO_QUEUED so that when a commit is
popped, it is produced if and only if TOPO_QUEUED is set.

To support nesting with other generators that may produce the same
commit multiple times like DepthGenerator (for example, StartGenerator
does this), do not increment parent inDegree for the same child commit
more than once.

Modify tests that assert that TopoSortGenerator mixes lines of commit
history.

Change-Id: I4ee03c7a8e5265d61230b2a01ae3858745b2432b
Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
2020-02-18 15:17:23 -08:00
Matthias Sohn bc98a597ff Update API problem filter
Change-Id: Iced4e3cc68cc9c2a75f9bf5befd6db7ce2fb1374
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-14 09:17:07 +01:00
Matthias Sohn 2899483e96 Prepare 5.6.2-SNAPSHOT builds
Change-Id: Ie04d749bf16eea6db5e5c98be54ba093ca249d05
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-14 00:15:39 +01:00
Matthias Sohn 1b44f0e4a6 JGit v5.6.1.202002131546-r
Change-Id: I2807b9a9f7e8b877b72702ebd1f70c4f9f223481
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-13 21:45:37 +01:00
Han-Wen Nienhuys 753fbaf11a Simplify ReftableCompactor
The ReftableCompactor supported a byteLimit, but this is currently
unused. The FileReftableStack has a more sophisticated strategy that
amortizes compaction costs.

Rename min/maxUpdateIndex to reflogExpire{Min,Max}UpdateIndex to
reflect their purpose  more accurately.

Since reflogs are generally pruned chronologically (oldest entries are
expired first), one can only prune entries on full compaction, so they
should not be set by default.

Rephrase the function Reader#minUpdateIndex and maxUpdateIndex. These
vars are documented to affect log entries, but semantically, they are
about ref entries. Since ref entries have their timestamps
delta-compressed, it is important for the min/maxUpdateIndex values to
be coherent between different tables.

The logical timestamps for log entries do not have to be coherent in
different tables, as the timestamps of a log entry is part of the key.
For example, a table written at update index 20 may contain a tombstone
log entry at timestamp 1.

Therefore, we set ReftableWriter's min/maxUpdateIndex from the merged
tables we are compacting, rather than from the compaction settings
(which should only control reflog expiry.)

The previous behavior could drop log entries erroneously, especially
in the presence of tombstone log entries. Unfortunately, testing this
properly requires both an API for adding log tombstones, and a more
refined API for controlling automatic compaction. Hence, no test.

Change-Id: I2f4eb7866f607fddd0629809e8e61f0b9097717f
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2020-02-12 23:08:20 +01:00
Yunjie Li 34cf825ba6 Change the wildcard import to explicit ones.
Change-Id: Ia7bc1c4b6b82c127e1e4e0e657f69d249684e8e6
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-02-12 18:45:26 +09:00
Matthias Sohn e68e0e7f03 Merge branch 'stable-5.6'
* stable-5.6:
  reftable: don't check deadline on the first try
  reftable: clarify comment
  reftable: clear cache on full compaction
  reftable: remove outdated comment
  reftable: clarify that LogCursor may return a null ReflogEntry

Change-Id: I9458a746311984fa687b3da964805e2568ed37f3
2020-02-11 01:39:27 +01:00
Han-Wen Nienhuys 79266a1fe5 reftable: don't check deadline on the first try
This helps debug stepping.

Change-Id: I020dafab4ffac75e6df0dbcde0ed4805c7867f72
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2020-02-11 00:03:33 +01:00
Han-Wen Nienhuys 936a031ca3 reftable: clarify comment
Change-Id: I16e32aeb325b604eb31f84db18a214f792395941
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 00:03:12 +01:00
Han-Wen Nienhuys 8c9f7656c3 reftable: clear cache on full compaction
The merged table contains handles to open files. A full compaction
causes those files to be closed, and so further lookups would fail
with EBADF.

Change-Id: I7bb74f7228ecc7fec9535b00e56a617a9c18e00e
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 00:03:09 +01:00
Han-Wen Nienhuys dd203f03c2 reftable: remove outdated comment
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I41a7d9934d53553b5f81f40ff9d663378676ac5d
2020-02-10 12:43:36 +01:00
Han-Wen Nienhuys 0bf2564433 reftable: clarify that LogCursor may return a null ReflogEntry
Change-Id: I1a4d5c262cd196dca37876aec00bb974a45e9fcd
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2020-02-10 12:43:36 +01:00
Matthias Sohn 009674e99f Merge branch 'stable-5.6'
* stable-5.6:
  Restore behavior of CloneCommand

Change-Id: I3092bf214c41436b57e0ede9d2202f8aabf15471
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-10 10:48:39 +01:00
Lars Vogel 9d1e477417 Use lambdas where possible
Using the batch cleanup operation via Source -> Cleanup -> "Use lambdas
where possible" from standard JDT.

Change-Id: I5452bd94fdccc920ade071228aeed3a8b9fdbe62
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
2020-02-05 06:11:06 -05:00
Thomas Wolf 75a80c5d3c Restore behavior of CloneCommand
Commit 6216b0de changed the behavior of the setMirror(),
setCloneAllBranches(), and setBranchesToClone() operations. Before
that commit, these could be set and reset independently and only in
call() it would be determined what exactly to do. Since that commit,
the last of these calls would determine the operation. This means
that the sequence

  cloneCommand.setCloneAllBranches(true);
  cloneCommand.setBranchesToClone(/* some list of refs */);

would formerly do a "clone all" giving a fetch refspec with wildcards

  +refs/heads/*:refs/remotes/origin/*

which picks up new upstream branches, whereas since commit 6216b0de
individual non-wildcard fetch refspecs would be generated and new
upstream branches would not be fetched anymore.

Undo this behavioral change. Make the operations independently settable
and resettable again, and determine the exact operation only in call():
mirror=true > cloneAll=true > specific refs, where ">" means "takes
precedence over", and if none is set assume cloneAll=true.

Note that mirror=true implies setBare(true).

Bug: 559796
Change-Id: I7162b60e99de5e3e512bf27ff4113f554c94f5a6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-02-04 08:49:07 +01:00
Joshua Redstone 2a134e1541 AmazonS3: Speed up fetch, try recent packs first
When fetching remote objects, WalkFetchConnection searches remote
packs in the order provided by WalkRemoteObjectDatabase:getPackNames.
Previously, for TransportAmazonS3, the packs were in no particular
order. This resulted in potential many extra calls to get pack idx
files.

This change modifies TransportAmazonS3 and AmazonS3 so that
getPackNames returns a list sorted with the most recently modified
packs first.  In the case of fetching recent changes to a repo,
this dramatically reduces the number of packs searched and speeds
up fetch.

Note: WalkRemoteObjectDatabase::getPackNames does not specify
the order of the returned names.

Testing: did "mvn clean install" in root dir and all tests passed.
And manually constructed some S3 repos and using jgit.sh
confirmed that the freshest pack was checked first.

Change-Id: I3b968fee825e793be55566e28c2d69d0cbe53807
Signed-off-by: Joshua Redstone <redstone@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-02 14:44:22 +01:00
Matthias Sohn eaff0a2eba Merge branch 'stable-5.7'
* stable-5.7:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I0538bcba259f7229790a602ac9de120464a1260d
2020-02-01 02:17:49 +01:00
Matthias Sohn 7e3e740cc7 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I79d615dff66493b60d3a4bcbdc57b9455e8d6673
2020-02-01 02:09:37 +01:00
Matthias Sohn 3d59d1b80c Merge branch 'stable-5.5' into stable-5.6
* stable-5.5:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I73d16b53df02bf735c2431588143efe225a4b5b4
2020-02-01 02:06:47 +01:00