Commit Graph

7948 Commits

Author SHA1 Message Date
Yunjie Li 840e414d0b Refactor: Make retriveCompressed an method of the Bitmap class
Make retrieveCompressed() a method of Bitmap interface to avoid type
casting and later reuse in improving the memory footprint of GC's bitmap
generation phase.

Change-Id: I098d85105cf17af845d43b8c71b4ca48b02fd7da
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:05 -07:00
Matthias Sohn a379d007db Fix downloading LFS Object fails behind proxy
When downloading LFS objects also accept response code 203 as successful
download. This response may be seen when downloading via a proxy.

Bug: 563022
Change-Id: Iee85fdb451b33369d08859872e5bfc2a67dffa6d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-12 10:26:45 -04:00
Terry Parker 0642e49f97 Merge changes I6543c2e1,I21ed029d
* changes:
  ReceivePack: adding IterativeConnectivityChecker
  Moving transport/internal -> internal/transport
2020-05-11 17:40:20 -04:00
Pat Long d27dceb384 Allow for using custom s3 host with lfs server
By default, it will generate hostname using the aws region passed to the
constructor.

This will allow for easier testing, since you can just spin up a local
minio (or other s3-compatible storage service) instance and point the
application at that for the storage mechanism.

It will also allow for storing lfs objects on-prem.

Change-Id: I2566b1fcce58f3d306ddd23a8da702ef5a451c7b
Signed-off-by: Pat Long <pllong@arista.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-10 11:29:27 +02:00
Demetr Starshov 9075beefb1 ReceivePack: adding IterativeConnectivityChecker
Introduce an IterativeConnectivityChecker which runs a connectivity
check with a filtered set of references, and falls back to using the
full set of advertised references.

It uses references during first check attempt:
- References that are ancestors of an incoming commits (e.g., pushing
a commit onto an existing branch or pushing a new branch based on
another branch)
- Additional list of references we know client can be interested in
(e.g. list of open changes for Gerrit)

We tested it inside Google and it improves connectivity for certain
topologies. For example connectivity counts for
chromium.googlesource.com/chromium/src:

percentile_50: 1923 (was: 22777)
percentile_90: 23272 (was: 353003)
percentile_99: 345522 (was: 353435)

This saved ~2 seconds on every push to this repository.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I6543c2e10ed04622ca795b195665133e690d3b10
2020-05-08 17:57:20 -07:00
Demetr Starshov 519cb1e91b Moving transport/internal -> internal/transport
Moving transport related internal classes into dedicated subpackage in
o/e/j/internal package.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I21ed029d359f5f7d8298f102efbb4b1dcdf404ad
2020-05-08 17:57:20 -07:00
Nail Samatov f30e48914e Fix error occurring during checkout
Fix NullPointerException occurring when calling
CheckoutCommand with forced == true option when
the branch isn't changed and there is deleted
uncommitted file.

Change-Id: I99bf1fc25e6889f07092320d7bc2772ec5d341b5
Signed-off-by: Nail Samatov <sanail@yandex.ru>
2020-05-07 12:18:44 +03:00
Matthias Sohn e69e642ea4 Merge branch 'stable-5.8'
* stable-5.8:
  Prepare 5.8.0-SNAPSHOT builds
  JGit v5.8.0.202005061305-m2

Change-Id: I7e95ddbedfae23035cc132abdf574545ffcda091
2020-05-07 00:33:49 +02:00
Matthias Sohn d1ea716bd0 Prepare 5.8.0-SNAPSHOT builds
Change-Id: Ic7d763b3b02b4d2a7a4141d2d9d715855dd69918
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-07 00:14:46 +02:00
Matthias Sohn 75fcccaea3 JGit v5.8.0.202005061305-m2
Change-Id: I9d9e5a9e000a74513896744cc3646430cb872893
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-06 19:06:11 +02:00
Matthias Sohn bcf4879781 Update to bouncycastle 1.65 and orbit I20200506000552
Bug: 553206
Change-Id: I867bbe88b847bf6d17f17f6cf8c565cd4a2437cc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-06 17:22:45 +02:00
Matthias Sohn 231c44d553 ApplyCommand: use Files#copy to copy file
This should be faster.

Change-Id: I404ec5e66731b3cf7a8e621cf1ff8748d109ea69
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-06 10:16:35 +02:00
Jack Wickham b6d70a66ae Apply hunks when renaming or copying from patch files
When applying a patch that contains renames or copies using ApplyCommand,
also apply all hunks that apply to the renamed or copied file.

Change-Id: I9f3fa4370458bd7c14beeb2e2b49e846d70203cb
Signed-off-by: Jack Wickham <jwickham@palantir.com>
2020-05-06 08:50:08 +02:00
Jack Wickham d69c0ef5bd Create parent directories when renaming a file in ApplyCommand
Before this change, applying a patch will fail if the destination directory
doesn't exist; after, the necessary parent directories are created.

If renaming the file fails, the directories won't be deleted, so this change
isn't atomic. However, ApplyCommand is already not atomic - if one hunk fails
to apply, other hunks still get applied - so I don't think that is a blocker.

Change-Id: Iea36138b806d4e7012176615bcc673756a82f365
Signed-off-by: Jack Wickham <jwickham@palantir.com>
2020-05-06 08:50:08 +02:00
Ivan Frade 0a2a094fea Reduce BitmappedObjectReachabilityChecker visibility
ObjectReachabilityChecker interface is the only public API. The
implementation is instantiated by ObjectWalk and doesn't need to be
visible outside the package.

Change-Id: I5b97bb98990cded637686bdc15c9655330b7780f
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-05-04 11:45:15 -07:00
Thomas Wolf 9f07528ff8 Add missing @since to new API
Change-Id: Ie7c3481aba515d9c9b0a152db17b5a9dc74e7ede
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-04 09:30:24 +02:00
David Ostrovsky 3a7e4eaecd Add missing test source file to the build
Change-Id: I020ff281eaa49794aeb1853711485bc8c7047dd8
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2020-05-03 23:20:18 +02:00
Michael Keppler 6fcd95cfed Upgrade Tycho to 1.7.0
Change-Id: I41d927e53f040750bbf0160dc15d8ca8a3f9f7ca
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
2020-05-01 10:34:22 +02:00
Ivan Frade 70ec9a83b8 ObjectReachabilityCheckers: Make walk member final
It is only assigned on initialization.

Change-Id: I36f68b3f511236e66cc7a4f56ba49252b317276a
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-30 10:23:29 -07:00
David Pursehouse 5072af6524 Upgrade wagon-ssh to 3.4.0
Change-Id: Id9a1bf7fcdba7c011ccff54188dfb1bef8bf8a4b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-04-29 18:34:07 -04:00
Matthias Sohn c67fbc122c Merge branch 'stable-5.7'
* stable-5.7:
  Bump Bazel version to 3.1.0

Change-Id: I108815577a5aab605891496c17d6f225d35f7346
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-29 23:22:51 +02:00
Matthias Sohn e624e13108 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  Bump Bazel version to 3.1.0

Change-Id: I8a9f39370ab506d9ff5763ed2d05f0d970bbd10b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-29 23:22:06 +02:00
Ivan Frade c9d7285e80 UploadPack: Use more relevant refs first in object reachability check
The bitmap-bassed object reachability checker, tries to find the objects
in the first starter, then adding the second starter... and so on. This
rewards passing the most popular refs first.

Order the refs with heads first, then tags, then others (e.g. changes)
for the object reachability checker. Using streams, delay also the
resolution of the ref to RevObject until necessary.

Change-Id: I9414b76754d7c0ffee1e2eeed6939895c8e92cbe
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade a661e2e9eb UploadPack: Refactor to generalize the object reachability checks
ObjectWalk#createObjectReachabilityChecker() returns the best
implementation for the repo. UploadPack can use the interface and fold
the with/without commits cases in one code path.

Change-Id: I857c11735d1d8e36c3ed8185ff11de8a62e86540
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade 6bc04bdc02 UploadPack: Use BitmappedReachabilityChecker for not advertised wants
Change-Id: Ifea971d5c0309e28a909441ee8a6f1e62397d6d3
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade 003002c1cb revwalk: Introduce bitmap-based object reachability checker
Change-Id: I0b1a2bd21f98894862aab339f8c2e4a417897b89
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:33 -07:00
David Pursehouse 916b367c27 Bump Bazel version to 3.1.0
Change-Id: Id58d1603846a88be3493165583adca481c026eed
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-04-29 13:53:45 +09:00
Ivan Frade 20bb312421 revwalk: Extract ObjectReachabilityChecker interface
Extract ObjectReachabilityChecker interface from the walk-based
implementation, to add a bitmapped based implementation later.

Refactor the test case to use it for both implementations.

Change-Id: Iaac7c6b037723811956ac22625f27d3b4d742139
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 16:56:32 -07:00
Ivan Frade ae26fa19b7 UploadPack: Extract walk-based reachability check
Preparing the code to optimize the bitmap-based object reachability
checker.  We are mirroring first the commit reachability checker
structure (interface + 2 implementations).

Move the walk-base reachability checker to its own class.

This class is public at the moment. Later ObjectWalk will return an
interface and this implementation will be package-private.

Change-Id: Ifac70094e1af137291c3607d95e689992f814b26
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 16:40:52 -07:00
Matthias Sohn df81c56524 Enable passing java options to jgit command line executable
This allows to pass java options to the jgit command line executable
by setting the environment variable "java_args", e.g.

$ java_args="-Xmx8g" jgit log

Change-Id: Ic7271b104737a8306d9db0f1895079b9f7bb7fd0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 13:57:42 +02:00
Matthias Sohn c845db3ec2 RefTreeBatch: fix unclosed resource warning
Change-Id: I7d630d11d3ef2ff2b74ec7a7018760b7b4931ad2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:48:14 +02:00
Matthias Sohn 7ae3507998 CherryPickCommand: fix unclosed resource warning
Change-Id: I2dc76efd16d887048a5cb4af2b2d8e6faa4f22cd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:48:14 +02:00
Matthias Sohn 560ef875e3 URIish: suppress non-localized message warning
Change-Id: I3ec37c67ba6f00ad8bf396aa3261dd90f35789ea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:21:06 +09:00
Matthias Sohn f3bc72fd21 Always use https to access download.eclipse.org
Change-Id: Ida3cd74dc80020308496f53bdda3676b616953d4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-27 14:52:04 +02:00
Minh Thai d9f84b0b7c UploadPack: Clear advertised ref map after negotiation
After negotiation phase of a fetch, the advertised ref map is no longer used and
can be safely cleared. For >1GiB repos object selection and packfile writing may
take 10s of minutes. For the chromium.googlesource.com/chromium/src repo, this
advertised ref map is >400MiB. Returning this memory to the Java heap is a major
scalability win.

Change-Id: I00d453c5ef47630c21f199e333e1cfcf47b7e92a
Signed-off-by: Minh Thai <mthai@google.com>
2020-04-23 12:14:02 -07:00
Lars Vogel a0802ff9c2 Use Map directly in MetaFilter
Change-Id: I15545e9c7cdcbb8a913cbcf0eef7f8a2281de7a8
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
2020-04-22 14:37:20 +02:00
Konrad Windszus 54a2d48008 Fix human name for local .bundle files
Bug: 560903
Change-Id: I15d45330398cc573940265d16a2db29ddce085aa
Signed-off-by: Konrad Windszus <konrad_w@gmx.de>
2020-04-21 07:46:50 +02:00
Matthias Sohn 7bebce42cc Merge branch 'stable-5.7'
* stable-5.7:
  Bazel: Disable SecurityManagerMissingPermissionsTest test

Change-Id: Id8665d1ba630c1359e86846548143a785b178e75
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-18 19:08:20 +02:00
David Ostrovsky 9b2f8ced90 Bazel: Disable SecurityManagerMissingPermissionsTest test
In Id5376f09f0d a test with dependency on log4j library was added, but
the library was missed to be added to the Bazel build tool chain.

Given that Bazel test runner doesn't suport custom security manager the
test wouldn't pass even if the missing dependency would be added. The
only solution we have for now is to exclude that test from Bazel tool
chain.

Filed a feature request for bazel to support such tests at
https://github.com/bazelbuild/bazel/issues/11146

Bug: 562274
Change-Id: I873a0e09addc583455b68122f66cd3952e485f0e
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-18 18:51:36 +02:00
Matthias Sohn 8ed7b26650 Merge branch 'stable-5.7'
* stable-5.7:
  Remove double blank from sentence start
  Bump Bazel version to 3.0.0
  Scan through all merged reftables for max/min update indices

Change-Id: Idee792c6e65ca3fb283dba850f17d6554aaa1fa7
2020-04-18 00:40:57 +02:00
Matthias Sohn a0aebbd528 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  Remove double blank from sentence start
  Bump Bazel version to 3.0.0

Change-Id: I26c3a8345020239d1c2ec5c6f70a633b43ddab86
2020-04-18 00:30:15 +02:00
Michael Keppler a79c5b1f10 Remove double blank from sentence start
Multiple whitespaces are not normalized when reading properties files,
therefore leading to unwanted space/indentation in console or UI output.

Change-Id: I1f5224fe359e0cac493e0237872afc75dc8b9fbe
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ebbc3efce7)
2020-04-17 14:36:50 +09:00
David Pursehouse f5be334a44 Upgrade maven-antrun-plugin to 3.0.0
Change-Id: I79d55d1c0b30cddfcc6a188af87ea08d802c4c3c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-04-16 20:15:45 +09:00
David Pursehouse 2b1d9d697a Upgrade maven-shade-plugin to 3.2.3
Change-Id: Idf84a1a22b250484db5d03cf55c5207be37511b9
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-04-16 20:15:45 +09:00
Michael Keppler ebbc3efce7 Remove double blank from sentence start
Multiple whitespaces are not normalized when reading properties files,
therefore leading to unwanted space/indentation in console or UI output.

Change-Id: I1f5224fe359e0cac493e0237872afc75dc8b9fbe
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
2020-04-14 09:31:50 +02:00
David Pursehouse 3851023e08 Bump Bazel version to 3.0.0
Change-Id: I3866b58d3d288f37b1818be5b56d05e8be6ecb0f
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-04-07 16:11:42 +09:00
Terry Parker 9c67f680c8 Merge "ReceivePack: Use error message if set" 2020-04-06 11:20:46 -04:00
Minh Thai 0eeed6ad2b Scan through all merged reftables for max/min update indices
Since reftables might have update index ranges that are overlapped.

Change-Id: I8f8215b99a0a978d4dd0155dbaf33e5e06ea8202
Signed-off-by: Minh Thai <mthai@google.com>
(cherry picked from commit 06748c205c)
2020-04-05 17:26:20 -04:00
Alexander Nittka bc4ed530a5 FileUtils: improve delete (Windows)
Ensure files are writable before trying to delete them.

Bug: 408846
Change-Id: I930a547594bba853c33634ae54bd64d236afade3
Signed-off-by: Alexander Nittka <alex@nittka.de>
2020-04-03 19:30:31 +02:00
Thomas Wolf 9aaa58052b Merge branch 'stable-5.7'
* stable-5.7:
  FS.runInShell(): handle quoted filters and hooksPath containing blanks
  Handle non-normalized index also for executable files
  Prepare 5.7.1-SNAPSHOT builds
  JGit v5.7.0.202003110725-r

Change-Id: I8a8580e44bfa05989d476cf22a029abd4fd407c6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-04-03 10:40:30 +02:00