Commit Graph

5798 Commits

Author SHA1 Message Date
Matthias Sohn e62a93955b Merge branch 'stable-5.4' into stable-5.5
* stable-5.4:
  Prepare 5.3.10-SNAPSHOT builds
  JGit v5.3.9.202012012026-r
  Prepare 5.1.16-SNAPSHOT builds
  JGit v5.1.15.202012011955-r
  Fix PackInvalidException when fetch and repack run concurrently

Change-Id: Iede8686198332d6271771bef6eb00c25f1103979
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 15:16:11 +01:00
Matthias Sohn e8948056fa Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  Prepare 5.3.10-SNAPSHOT builds
  JGit v5.3.9.202012012026-r
  Prepare 5.1.16-SNAPSHOT builds
  JGit v5.1.15.202012011955-r
  Fix PackInvalidException when fetch and repack run concurrently

Change-Id: I319b54ceffe095add60420c6ae83eac0ba9c14b6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 14:26:37 +01:00
Matthias Sohn 0a63e045c9 Prepare 5.3.10-SNAPSHOT builds
Change-Id: I6f131ad04574bd0d569ae6a59c29ea987be0efb2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 13:09:57 +01:00
Matthias Sohn aa5d1e425c JGit v5.3.9.202012012026-r
Change-Id: I75e03ef8630d7a369e97be0f797253b968575354
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 02:27:44 +01:00
Matthias Sohn 5bb71dd2d4 Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Prepare 5.1.16-SNAPSHOT builds
  JGit v5.1.15.202012011955-r
  Fix PackInvalidException when fetch and repack run concurrently

Change-Id: If1d8034b5e0cbc004a11a31b228cb5732efb390d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 02:19:11 +01:00
Matthias Sohn babc56ef0f Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Prepare 5.1.16-SNAPSHOT builds
  JGit v5.1.15.202012011955-r
  Fix PackInvalidException when fetch and repack run concurrently

Change-Id: Id83e29e567646a3945a5b817860ea8f7c3e6e5cf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 02:17:41 +01:00
Matthias Sohn 5e0cfce5ad Prepare 5.1.16-SNAPSHOT builds
Change-Id: I50e59e1e73a92fa4fe366398fb8141f5e2e289c1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 02:02:48 +01:00
Matthias Sohn f72a001250 JGit v5.1.15.202012011955-r
Change-Id: Icb4f04a40ab366cbacbb3fdf0db1748f27277fda
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 01:53:39 +01:00
Petr Hrebejk 2fbbd6d890 Fix PackInvalidException when fetch and repack run concurrently
We are running several servers with jGit. We need to run repack from
time to time to keep the repos performant. I.e. after push we test how
many small packs are in the repo and when a threshold is reached we run
the repack.

After upgrading jGit version we've found that if someone does the clone
at the time repack is running the clone sometimes (not always) fails
because the repack removes .pack file used by the clone. Server
exception and client error attached.

I've tracked down the cause and it seems to be introduced between jGit
5.2 (which we upgraded from) and 5.3 and being caused by this commit:
Move throw of PackInvalidException outside the catch -
afef866a44

The problem is that when the throw was inside of the try block the last
catch block catched the exception and called openFailed(false) method.
It is true that it called it with invalidate = false, which is wrong.
The real problem though is that with the throw outside of the try block
the openFail is not called at all and the fields activeWindows and
activeCopyRawData are not set to 0. Which affects the later called tests
like: if (++activeCopyRawData == 1 && activeWindows == 0).

The fix for this is relatively simple keeping the throw outside of the
try block and still having the invalid field set to true. I did
exhaustive testing of the change running concurrent clones and pushes
indefinitely and with the patch applied it never fails while without the
patch it takes relatively short to get the error.

See: https://www.eclipse.org/lists/jgit-dev/msg04014.html

Bug: 569349
Change-Id: I9dbf8801c8d3131955ad7124f42b62095d96da54
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-12-02 01:43:25 +01:00
Matthias Sohn 180dfdefa4 Merge branch 'stable-5.9'
* stable-5.9:
  Add constants for parsing git wire protocol version

Change-Id: I1e4174ae8bbefcb71a191dcbf32027d4b35c2b37
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-30 16:54:02 +01:00
David Ostrovsky 23389a6323 Add constants for parsing git wire protocol version
This would allow other JGit users to access and reuse the constants.

Change-Id: I1608802f45586af5f8582afa592e26679e9cebe3
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-29 22:46:51 +01:00
Matthias Sohn 4f2065d145 Merge branch 'stable-5.10'
* stable-5.10:
  Remove unused imports
  Silence API warnings
  Remove erraneously merged source features
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  PacketLineIn: ensure that END != DELIM
  Update API warning filters
  Remove unused imports

Change-Id: I25f50c3807a4e6b22a264320ea7ed3758e2a75ec
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 10:59:01 +01:00
Matthias Sohn d4fe93f55b Merge branch 'stable-5.9' into stable-5.10
* stable-5.9:
  Remove unused imports
  Silence API warnings
  Remove erraneously merged source features
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  PacketLineIn: ensure that END != DELIM
  Update API warning filters
  Remove unused imports

Change-Id: Icf415ac5bab2f69f71189c942424ee69f8a64d4e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 10:10:44 +01:00
Matthias Sohn 5cd485e5dd Merge branch 'stable-5.8' into stable-5.9
* stable-5.8:
  Remove unused imports
  Silence API warnings
  Remove erraneously merged source features
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  PacketLineIn: ensure that END != DELIM
  Update API warning filters
  Remove unused imports

Change-Id: I70b399eb3df02aa2fb112e86e844cf5a0daa5515
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 09:38:42 +01:00
Matthias Sohn d1801402fe Merge branch 'stable-5.7' into stable-5.8
* stable-5.7:
  Remove unused imports
  Silence API warnings
  Remove erraneously merged source features
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  Update API warning filters
  Remove unused imports

Change-Id: I9b94938f5c09bd726e8e368c98c56da8280fd0b2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 09:07:30 +01:00
Matthias Sohn f65271f843 Silence API warnings
Change-Id: I5de476935e79f8f0beefc27885cbc3ffe31c0aed
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 02:26:23 +01:00
Matthias Sohn 99f5329c38 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  Update API warning filters
  Remove unused imports

Change-Id: Ie24d381f295cccfb99068c7ed5817179da29c1db
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 01:03:02 +01:00
Matthias Sohn 480b00f1c7 Merge branch 'stable-5.5' into stable-5.6
* stable-5.5:
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  Update API warning filters
  Remove unused imports

Change-Id: I00c8ddad0059a20a978743dfb7ad1b513dc7fef6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 01:01:31 +01:00
Matthias Sohn 95846863cb Merge branch 'stable-5.4' into stable-5.5
* stable-5.4:
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  Update API warning filters
  Remove unused imports

Change-Id: Icc34f809a3bb019d8d640b9bdb71363e617942e2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 00:40:26 +01:00
Matthias Sohn a74c0cd333 Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  Prepare 5.3.9-SNAPSHOT builds
  JGit v5.3.8.202011260953-r
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  Update API warning filters
  Remove unused imports

Change-Id: I82576ad0f61cf3ff11b54691b32666c61401ad9b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-27 00:29:53 +01:00
Lee Worrall 9ebbfe93bb Add support for reading symrefs from pack capabilities
A SymbolicRef is added to the advertised refs for any symref in
capabilities whose target is an advertised ref; this may replace an
existing entry, such as HEAD.

When cloning, if any advertised HEAD is symbolic then use the target
rather than looking for an advertised ref with a matching objectId.

Add --symref option to LsRemote command.

Bug: 514052
Change-Id: Idfb48e6f6e8dcfe57a6896883fe6d84d533aa9d0
Signed-off-by: Lee Worrall <worrall.la@gmail.com>
2020-11-26 21:14:08 +01:00
Matthias Sohn fdcdfc37a3 Prepare 5.3.9-SNAPSHOT builds
Change-Id: I69f181453c79cef2b4f43fac38d9836917cdc973
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 16:30:32 +01:00
Matthias Sohn 0e9cf495d9 JGit v5.3.8.202011260953-r
Change-Id: I7b128942ef224335f415f867c2d9d5da7498ed8b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 15:52:53 +01:00
Matthias Sohn 3f85d3b75c Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  Update API warning filters
  Remove unused imports

Change-Id: Id9386846a202b5ae98dd602744963f8897ddaa8c
2020-11-26 15:37:19 +01:00
Matthias Sohn 65598b9516 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Prepare 5.1.15-SNAPSHOT builds
  JGit v5.1.14.202011251942-r
  GC#deleteOrphans: log warning for deleted orphaned files
  GC#deleteOrphans: handle failure to list files in pack directory
  Ensure that GC#deleteOrphans respects pack lock
  Update API warning filters
  Remove unused imports

Change-Id: I91cfe2820c40d2d773cbf018cc2a6c36b062801e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 15:34:24 +01:00
Matthias Sohn 8c2dc8b6c0 Prepare 5.1.15-SNAPSHOT builds
Change-Id: I70246b66c76e865aef4e3adada3a507750ca7c63
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 15:02:01 +01:00
Matthias Sohn dac540a37f JGit v5.1.14.202011251942-r
Change-Id: Ibe124988be39feaa029c3770777126dd87b18abc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 01:42:17 +01:00
Matthias Sohn 0f1735de0b GC#deleteOrphans: log warning for deleted orphaned files
Change-Id: Ie245bf5c8c924dfb1f0f40b8bcdcb1e6f5815526
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 01:34:42 +01:00
Matthias Sohn 3996408cde GC#deleteOrphans: handle failure to list files in pack directory
- log an error
- either there is no list or it is incomplete hence return immediately

Change-Id: Ieee5378ca06304056b9ccc30c1acd5f52360052d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 01:17:49 +01:00
Matthias Sohn fde7a271a4 Ensure that GC#deleteOrphans respects pack lock
If pack or index files are guarded by a pack lock (.keep file)
deleteOrphans() should not touch the respective files protected by the
lock file. Otherwise it may interfere with PackInserter concurrently
inserting a new pack file and its index.

The problem was caused by the following race.

All mentioned files are located in  "objects/pack/".
File endings relevant in "pack" dir:
  .pack
  .keep
  .idx
  .bitmap

When ReceivePack receives a pack file it executes the following steps:

ReceivePack.service():
  receivePackAndCheckConnectivity():
    receivePack():
      receive the pack
      parse the pack, returns packLock (.keep file)
      PackInserter.flush():
        write tmpPck file:  "insert_<random>.pack"
        write tmpIdx file:  "insert_<random>.idx"
        real pack name: "pack-<SHA1>.pack"
        real index name: "pack-<SHA1>.idx"
        atomic rename tmpPack to realPack
        atomic rename tmpIdx to tmpIdx
  execute commands
  unlock pack by removing .keep file
  trigger auto gc if enabled

When PackInserter.flush() renames the temporary pack to the final
"pack-xxx.pack" file  the temporary pack index file "insert_xxx.idx"
has no matching .pack file with the same base name for a short interval.
If deleteOrphans() ran during that interval it deduced the pack index
file was orphaned. Subsequently the missing pack index caused
MissingObjectExceptions since objects contained in the pack couldn't be
looked up anymore.

Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=13544
Change-Id: I559c81e4b1d7c487f92a751bd78b987d32c98719
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-26 01:17:47 +01:00
Matthias Sohn 63f16c53c8 Prepare 5.10.0-SNAPSHOT builds
Change-Id: Iabc845edf565ac706344f68f854384faa63b6f43
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-25 22:30:34 +01:00
Matthias Sohn 2486be2706 JGit v5.10.0.202011251205-m3
Change-Id: Id58d8eb49f550e62b04053a71366f7eabd9f9a53
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-25 18:05:35 +01:00
Thomas Wolf 9f3616dcb4 PacketLineIn: ensure that END != DELIM
Just allocate the two string objects directly. The previously used
new StringBuilder(0).toString() returns the same object for both END
and DELIM when run on Java 15, which breaks the wire protocol since
then END and DELIM cannot be distinguished anymore.

Bug: 568950
Change-Id: I9d54d7bf484948c24b51a094256bd9d38b085f35
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
(cherry picked from commit 7da0f0a8f3)
2020-11-20 23:30:24 +01:00
Thomas Wolf e2663a8b85 PacketLineIn: ensure that END != DELIM
Just allocate the two string objects directly. The previously used
new StringBuilder(0).toString() returns the same object for both END
and DELIM when run on Java 15, which breaks the wire protocol since
then END and DELIM cannot be distinguished anymore.

Bug: 568950
Change-Id: I9d54d7bf484948c24b51a094256bd9d38b085f35
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
(cherry picked from commit 7da0f0a8f3)
2020-11-19 21:07:25 +01:00
Thomas Wolf 7da0f0a8f3 PacketLineIn: ensure that END != DELIM
Just allocate the two string objects directly. The previously used
new StringBuilder(0).toString() returns the same object for both END
and DELIM when run on Java 15, which breaks the wire protocol since
then END and DELIM cannot be distinguished anymore.

Bug: 568950
Change-Id: I9d54d7bf484948c24b51a094256bd9d38b085f35
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-11-19 13:23:05 +01:00
Thomas Wolf e84881ea6b Allow to resolve a conflict by checking out a file
DirCacheEditor unconditionally applied a PathEdit to all stages in the
index. This gives wrong results if one wants to check out a file from
some commit to resolve a conflict: JGit would update the working tree
file multiple times (once per stage), and set all stages to point to
the checked-out blob.

C git replaces the stages by the entry for the checked-out file.

To support this, add a DirCacheEntry.setStage() method so that
CheckoutCommand can force the stage to zero. In DirCacheEditor, keep
only the zero stage if the PathEdit re-set the stage.

Bug: 568038
Change-Id: Ic7c635bb5aaa06ffaaeed50bc5e45702c56fc6d1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-11-17 14:27:28 +01:00
Matthias Sohn f39f848c99 Document that setLastModified sets time of symlink target
Due to Java bug JDK-8220793 [1] Java cannot set timestamps of a symlink
but only of the symlink target. This bug was fixed in Java 13. Since we
don't have a use case to set the timestamp of the symlink itself simply
document the current behavior of setLastModified methods.


[1] https://bugs.openjdk.java.net/browse/JDK-8220793

Change-Id: Ibc28c1702a1b9845602bd257606fbd44803a43fb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-07 03:53:13 -05:00
Alexa Panfil 4f3161d3cc Fix bug in PerformanceLogContext
PerformanceLogContext threw NullPointerException when multiple threads
tried to add an event to the PerformanceLogContext. The cause for this
is that the ThreadLocal was initialized only in the class constructor
for the first thread; for subsequent threads it was null.

To fix this initialize eventRecords for each thread.

Change-Id: I18ef67dff8f0488e3ad28c9bbc18ce73d5168cf9
Signed-off-by: Alexa Panfil <alexapizza@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-06 19:20:08 -04:00
Nail Samatov d76088bca6 Fix IOException occurring during gc
Fix IOException occurring when calling
GC on a repository with absent objects/pack folder.

Change-Id: I5be1333a0726f4d7491afd25ddac85451686c30a
Signed-off-by: Nail Samatov <sanail@yandex.ru>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-11-05 01:31:14 +01:00
Thomas Wolf d69fb4d4ac Revert "Client-side protocol V2 support for fetching"
This reverts commit f802f06e7f.

I had misunderstood how protocol V2 works. This implementation only
works if the negotiation during fetch is done in one round.

Fixing this is substantial work in BasePackFetchConnection. Basically
I think I'd have to change back negotiate to the V0 version, and have
a doFetch() that does

  if protocol V2
    doFetchV2()
  else
    doFetchV0()

with doFetchV0 the old code, and doFetchV2 completely new.

Plus there would need to be a HTTP test case requiring several
negotiation rounds.

This is a couple of days work at least, and I don't know when I will
have the time to revisit this. So although the rest of the code is
fine I prefer to back this out completely and not leave a only half
working implementation in the code for an indeterminate time.

Bug: 553083
Change-Id: Icbbbb09882b3b83f9897deac4a06d5f8dc99d84e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-11-03 23:50:21 +01:00
Thomas Wolf f802f06e7f Client-side protocol V2 support for fetching
Make all transports request protocol V2 when fetching. Depending on
the transport, set the GIT_PROTOCOL environment variable (file and
ssh), pass the Git-Protocol header (http), or set the hidden
"\0version=2\0" (git anon). We'll fall back to V0 if the server
doesn't reply with a version 2 answer.

A user can control which protocol the client requests via the git
config protocol.version; if not set, JGit requests protocol V2 for
fetching. Pushing always uses protocol V0 still.

In the API, there is only a new Transport.openFetch() version that
takes a collection of RefSpecs plus additional patterns to construct
the Ref prefixes for the "ls-refs" command in protocol V2. If none
are given, the server will still advertise all refs, even in protocol
V2.

BasePackConnection.readAdvertisedRefs() handles falling back to
protocol V0. It newly returns true if V0 was used and the advertised
refs were read, and false if V2 is used and an explicit "ls-refs" is
needed. (This can't be done transparently inside readAdvertisedRefs()
because a "stateless RPC" transport like TransportHttp may need to
open a new connection for writing.)

BasePackFetchConnection implements the changes needed for the protocol
V2 "fetch" command (simplified ACK handling, delimiters, section
headers).

In TransportHttp, change readSmartHeaders() to also recognize the
"version 2" packet line as a valid smart server indication.

Adapt tests, and run all the HTTP tests not only with both HTTP
connection factories (JDK and Apache HttpClient) but also with both
protocol V0 and V2. Do the same for the SSH transport tests.

Bug: 553083
Change-Id: Ice9866aa78020f5ca8f397cde84dc224bf5d41b4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-10-29 00:36:21 +01:00
John Dallaway f4b4dae2be Ensure .gitmodules is loaded when accessing submodule name
This problem occurred when calling SubmoduleWalk#getModuleName if the
first submodule processed has a name and a path which do not match
SubmoduleWalk#getModuleName returned the module path instead of the
module name. In order to fix this SubmoduleWalk#getModuleName needs to
ensure that the modules config is loaded.

Bug: 565776
Change-Id: I36ce1fbc64c4849f9d8e39864b825c6e28d344f8
Signed-off-by: John Dallaway <john@dallaway.org.uk>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-10-25 22:24:17 +01:00
Matthias Sohn 8269587f49 Export new package org.eclipse.jgit.logging and import it where used
Change-Id: Ib81b8733e184e75c2cba21ab27c4bbe8dbe0b04d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-10-25 15:45:49 +01:00
Matthias Sohn e23bdf1c7c Ensure GC.deleteOrphans() can delete read-only orphaned files on Windows
Reported in https://www.eclipse.org/lists/jgit-dev/msg04005.html

Change-Id: I140a49a7f65a76aa2b67ec8d286a3d2506ae499a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-10-22 09:25:55 +02:00
Terry Parker dd593205a0 Merge "Add new performance logging" 2020-10-21 11:53:35 -04:00
Alexa Panfil a09e205176 Add new performance logging
Add new performance logging to register events of type duration.
The proposed logging is similar to the performance logging
in OS Gerrit https://gerrit-review.googlesource.com/c/gerrit/+/225628:
a global Singleton (LoggingContext in Gerrit) is
collecting the performance logs in a thread-safe events list,
and at the end of the monitored command the list of events is
retrieved and written to a log, after which it is cleared.

What this patch does:
The main component is the Singleton (PerformanceLogContext), which
is used to collect the records (PerformanceLogRecord) in one place
(ThreadLocal eventsList) from anywhere using
PerformanceLogContext.getInstance().addEvent().

Reason why this change is needed:
The current monitoring in JGit has several issues:
1. git fetch and git push events are handled separately
(PackStatistics and ReceivedPackStatistics), with no unified way
of writing or reading the statistics.
2. PostUploadHook is only invoked on the event of sending the
pack, which means that the PackStatistics is not available for
the fetch requests that did not end with sending the pack
(negotiation requests).
3. The way the logs are created is different from the performance
log approach, so the long-running operations need to be collected
from both performance log (for JGit DFS overridden operations and
Gerrit operations) and gitlog (for JGit ones).

The proposed performance logging is going to solve the above
mentioned issues: it collects all of the performance logs in one
place, thus accounting for the commands that do not result in
sending a pack. The logs are compatible with the ones on Gerrit.
Moreover, the Singleton is accessible anywhere in the call stack,
which proved to be successful in other projects like Dapper
(https://research.google/pubs/pub36356/).

Signed-off-by: Alexa Panfil <alexapizza@google.com>
Change-Id: Iabfe667a3412d8a9db94aabb0f39b57f43469c41
2020-10-21 12:54:30 +00:00
Jonathan Tan de914b7caf Merge "Compute time differences with Duration" 2020-10-19 12:25:57 -04:00
Christian Halstrick 88e924e86b Merge "Implement git describe --all" 2020-10-13 08:14:14 -04:00
Jason Yeo 276fcb2a11 Implement git describe --all
This enables jgit to use any refs in the refs/ namespace when describing
commits.

Signed-off-by: Jason Yeo <jasonyeo88@gmail.com>
Change-Id: I1fa22d1c39c0e2f5e4c2938c9751d8556494ac26
2020-10-13 18:06:39 +08:00
Alexa Panfil a0d3680f49 Compute time differences with Duration
Reason why this change is needed:
Currently the durations of fetch events are computed by
registering time instants with System.currentTimeMillis()
and calculating the differences with simple minus operation,
but multiple sources suggest that the best practice is to use
the Java 8 Duration and Instant objects.

What this patch does:
Get time measurements with Instant.now() instead of
System.currentTimeMillis() and calculate the duration of fetch
events (Reachability checks and Negotiation) using
Duration.between().toMillis().

Signed-off-by: Alexa Panfil <alexapizza@google.com>
Change-Id: I573a0a0562070083cf5a5a196d9710f69a7fa587
2020-10-09 18:55:20 +00:00
Thomas Wolf f37aa182e1 Override config http.userAgent from environment GIT_HTTP_USER_AGENT
According to [1], environment variable GIT_HTTP_USER_AGENT can
override a git config http.userAgent.

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpuserAgent

Change-Id: I996789dc49faf96339cd7b4e0a682b9bcafb6f70
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-10-06 19:03:36 -04:00
David Ostrovsky f3bb71206c Fix OperatorPrecedence warning flagged by error prone
Building with Bazel is failing with this error message:

HttpSupport.java:480: error: [OperatorPrecedence] Use grouping
parenthesis to make the operator precedence explicit
    if (c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z') {
                             ^
    (see https://errorprone.info/bugpattern/OperatorPrecedence)
Did you mean 'if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')) {'?

Change-Id: I96089d3158d06ba981cfdf2b03865261b328de23
2020-10-02 18:52:10 +02:00
Matthias Sohn 7fbc0e02a0 ObjectDirectory#selectObjectRepresentation: fix formatting
Change-Id: I3872f3001bb11e29a526ed90184ccb0c991b8567
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-10-02 00:55:04 +02:00
James Wynn 2171f868d9 Support "http.userAgent" and "http.extraHeader" from the git config
Validate the extra headers and log but otherwise ignore invalid
headers. An empty http.extraHeader starts the list afresh.

The http.userAgent is restricted to printable 7-bit ASCII, other
characters are replaced by '.'.

Moves a support method from the ssh.apache bundle to HttpSupport in
the main JGit bundle.

Bug:541500
Change-Id: Id2d8df12914e2cdbd936ff00dc824d8f871bd580
Signed-off-by: James Wynn <james@jameswynn.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-09-26 23:42:38 +02:00
Thomas Wolf e21d7b2d3c API filters for PackStatistics.Accumulator
Change-Id: Iaafb766aa76f0e8f837bd37c0729b1262e7ac1f6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-09-26 19:56:46 +02:00
Thomas Wolf 6f41458d8a Add TypedConfigGetter.getPath()
This enables EGit to override with a lenient variant that logs the
problem and continues with the default value. EGit needs this because
otherwise a corrupt core.excludesFile entry can render the whole UI
unusable (until the git config is fixed) because any use of the
WorkingTreeIterator will throw an InvalidPathException.

This is not a problem on OS X, where all characters are allowed in
file names. But on Windows some characters are forbidden... see bug
567296. The message of the InvalidPathException is not helpful since
it doesn't point to the origin of the problem. EGit can log a much
better message indicating the offending config file and entry in the
Eclipse error log, where the user can see it.

Bug: 567309
Change-Id: I4e57afa715ff3aaa52cd04b5733f69e53af5b1e0
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-09-25 18:40:10 -04:00
Alexa Panfil 597a6bc7ec Make Javadoc consistent for PackStatistics fields
Change-Id: I53d4072174e9fec92bff7c0225aa7470d8d39d05
2020-09-24 19:41:55 +00:00
Alexa Panfil 3c5e159eae Measure time taken for reachability checks
Reason why this change is needed:
Getting this metric will help estimate how much time will be saved once
the reachability checks get optimized

What this patch does:
Measure time spent by requestValidator.checkWants() in parseWants() and save
it in an instance of PackStatistics.Accumulator.

Signed-off-by: Alexa Panfil <alexapizza@google.com>
Change-Id: Id7fe4016f96549d9511a2c24052dad93cfbb31a4
2020-09-24 12:32:02 +00:00
Terry Parker be403859c1 Merge "Measure time taken for negotiation in protocol V2" 2020-09-22 11:34:24 -04:00
Alexa Panfil c3458a6e03 Measure time taken for negotiation in protocol V2
Reason why this change is needed:
Getting this metric will help estimate how much time is spent
on negotiation in fetch V2.

What this patch does:
Measure time spent on negotiation rounds in UploadPack.fetchV2()
and save it in an instance of PackStatistics.Accumulator.
This is the same way the statistics are already gathered for
protocol V0/V1.

Change-Id: I14e55dd6ff743cb0b21b4953b533269ef069abb1
2020-09-22 10:05:03 +00:00
Thomas Wolf cb553e3583 IndexDiffFilter: handle path prefixes correctly
When comparing git directory paths to check whether one is a prefix
of another, one must add a slash to avoid false prefix matches when
one directory name is a prefix of another. The path "audio" is not
a prefix of the path "audio-new", but would be a prefix of a path
"audio/new".

Bug: 566799
Change-Id: I6f671ca043c7c2c6044eb05a71dc8cca8d0ee040
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-09-21 16:52:43 -04:00
Thomas Wolf 566e49d7d3 sshd: support the ProxyJump ssh config
This is useful to access git repositories behind a bastion server
(jump host).

Add a constant for the config; rewrite the whole connection initiation
to parse the value and (recursively) set up the chain of hops. Add
tests for a single hop and two different ways to configure a two-hop
chain.

The connection timeout applies to each hop in the chain individually.

Change-Id: Idd25af95aa2ec5367404587e4e530b0663c03665
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-09-19 15:17:00 -04:00
Terry Parker 292919b12a Merge "ReceivePackStats: Add size and count of unnecessary pushed objects" 2020-09-14 09:53:43 -04:00
Yunjie Li 58e991b5de ReceivePackStats: Add size and count of unnecessary pushed objects
Since there is no negotiation for a push, the client is probably sending
redundant objects and bytes which already exist in the server.

Add more metrics in the stats to quantify it. Duplicated size and number
to measure the size and the number of duplicated objects which should
not be pushed.

Change-Id: Iaacd4761ee9366a0a7ec4e26c508eff45c8744de
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-09-11 16:19:57 -07:00
Matthias Sohn 8cd49885ba Merge branch 'stable-5.9' into master
* stable-5.9:
  Prepare 5.9.1-SNAPSHOT builds
  JGit v5.9.0.202009080501-r
  [releng] Enable japicmp for the fragments added in 5.8.0
  GitlinkMergeTest: fix boxing warnings
  Remove unused API problem filters
  Add missing since tag on BundleWriter#addObjectsAsIs
  GPG: include signer's user ID in the signature

Change-Id: Iaa96f9228752540f446fc232a49f31a738fd8d30
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-09 00:14:08 +02:00
Matthias Sohn ed9992896a Prepare 5.9.1-SNAPSHOT builds
Change-Id: I9006e7961111982943ffef496d15bd525959b3e4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-08 17:45:20 +02:00
Matthias Sohn dd169769bf JGit v5.9.0.202009080501-r
Change-Id: Ic98ae61b3f327ef72256fd9b2e28510e3a481de7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-08 13:01:43 +02:00
Matthias Sohn 8bb172acdb Remove unused API problem filters
Change-Id: Ice154340e3377e89dc51d40587c8a4c5caec2bcf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-05 23:25:58 +02:00
Matthias Sohn 94aab2f634 Add missing since tag on BundleWriter#addObjectsAsIs
Change-Id: Ia07c0ede31e280831869db77b70b43874d8ba3ac
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-05 23:25:02 +02:00
Matthias Sohn daddfe051b Merge branch 'master' into stable-5.9
* master:
  SshdSession: close channel gracefully
  jgit: Add DfsBundleWriter
  Prepare 5.10.0-SNAPSHOT builds
  ResolveMerger: do not content-merge gitlinks on del/mod conflicts
  ResolveMerger: Adding test cases for GITLINK deletion
  ResolveMerger: choose OURS on gitlink when ignoreConflicts
  ResolveMerger: improving content merge readability
  ResolveMerger: extracting createGitLinksMergeResult method
  ResolveMerger: Adding test cases for GITLINK merge

Back out the version change to 5.10.0-SNAPSHOT which was done on master
already.

Change-Id: I1a6b1f0b8f5773be47823d74f593d13b16a601d5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-05 22:51:34 +02:00
Masaya Suzuki 9d2055152c jgit: Add DfsBundleWriter
DfsBundleWriter writes out the entire repository to a Git bundle file.
It packs all objects included in the packfile by concatenating all pack
files. This makes the bundle creation fast and cheap. Useful for backing
up a repository as-is.

Change-Id: Iee20e4b1ab45b2a178dde8c72093c0dd83f04805
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2020-09-03 22:58:37 +00:00
Terry Parker 957419610a Merge changes from topic "fix_ui"
* changes:
  ResolveMerger: do not content-merge gitlinks on del/mod conflicts
  ResolveMerger: Adding test cases for GITLINK deletion
  ResolveMerger: choose OURS on gitlink when ignoreConflicts
  ResolveMerger: improving content merge readability
  ResolveMerger: extracting createGitLinksMergeResult method
  ResolveMerger: Adding test cases for GITLINK merge
2020-09-03 18:35:24 -04:00
Marco Miller df6a1c55fa Merge branch 'stable-5.9'
* stable-5.9:
  Upgrade maven-resources-plugin to 3.2.0
  Upgrade plexus-compiler version to 2.8.8
  [bazel] Add missing dependency to slf4j-api
  [errorprone] DirCacheEntry: make clear operator precedence
  [errorprone] PackWriter#parallelDeltaSearch: avoid suppressed exception
  [errorprone] Declare DirCache#version final
  Add jgit-4.17-staging target platform for 2020-09
  Update target platform to R20200831200620

Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Change-Id: I2e2f41cf6ebbcb45b8978b519db3f1c8f7afb5f4
2020-09-02 16:50:05 -04:00
Matthias Sohn 50e81a52c0 [errorprone] DirCacheEntry: make clear operator precedence
See https://errorprone.info/bugpattern/OperatorPrecedence

Change-Id: I96f2c844a7b3d9d5605122118d9420bde31ec2f0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-01 17:51:45 +02:00
Matthias Sohn 1d95f9cb8f [errorprone] PackWriter#parallelDeltaSearch: avoid suppressed exception
See https://errorprone.info/bugpattern/Finally

Change-Id: Ic2ad0d1e1ba7552b5a5c6238f83c0e13a94254d0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-01 17:51:45 +02:00
Matthias Sohn 04622016ea [errorprone] Declare DirCache#version final
Enums values should be immutable, see
https://errorprone.info/bugpattern/ImmutableEnumChecker.

Change-Id: Ib0a358d3a5f1560ca73ec3153ca8088fe7a35eb6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-09-01 17:51:44 +02:00
Matthias Sohn 6ae1a30004 Prepare 5.10.0-SNAPSHOT builds
Change-Id: I9a2b39e9e85f27179ceb3b1709d75c466089a3bc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-08-27 16:14:36 +02:00
Matthias Sohn a3d7240352 Prepare 5.9.0-SNAPSHOT builds
Change-Id: Ia3e8382ec503150979d8acb6161031ccfb7fd921
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-08-27 10:22:51 +02:00
Demetr Starshov 214c4afc2c ResolveMerger: do not content-merge gitlinks on del/mod conflicts
Previously ResolveMerger tried to make a fulltext merge entry in case
one of sides got deleted regardless of file mode. This is not
applicable for GITLINK type of entry. After this change it is
rendering appropriate merge result.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: Ibdb4557bf8781bdb48bcee6529e37dc80582ed7e
2020-08-26 18:40:28 -07:00
Demetr Starshov c084729f79 ResolveMerger: choose OURS on gitlink when ignoreConflicts
Option ignoreConflicts is used when a caller want to create a virtual
commit and use it in a future merge (recursive merge) or show it on
UI (e.g. Gerrit). According to contract in case of ignoreConflicts
ResolveMerger should populate only stage 0 for entries with merge
conflicts as if there is no conflict. Current implementation breaks
this contract for cases when gitlink revision is ambiguous.

Therefore, always select 'ours' when we merge in ignoreConflicts mode.
This will satisfy the contract contract, so recursive merge can
succeed, however it is an arbitrary decision, so it is not guaranteed
to select best GITLINK in all cases.

GITLINK merging is a special case of recursive merge because of
limitations of GITLINK type of entry. It can't contain more than 1 sha-1
so jgit can't write merge conflicts in place like it can with a blob.
Ideally we could signal the conflict with a special value (like
'0000...'), but that must be supported by all tooling (git fsck, c-git)."

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: Id4e9bebc8e828f7a1ef9f83259159137df477d89
2020-08-26 18:39:48 -07:00
Demetr Starshov 29e998a0e6 ResolveMerger: improving content merge readability
Separate "GITLINK conflict" and "attributes can't be content merged"
cases.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I29424e13ea1738af750196e7bf4315256a6095b6
2020-08-26 18:39:48 -07:00
Demetr Starshov 3da7ea50a9 ResolveMerger: extracting createGitLinksMergeResult method
Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: Ibc8b954266b1b4b9b9f404e3433f0d7cdae107e8
2020-08-26 18:39:48 -07:00
Matthias Sohn bf6b2b9314 JGit v5.9.0.202008260805-m3
Change-Id: Ic4de8340f3ab038e38b239b725b8bd6d6dbee413
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-08-26 14:05:07 +02:00
Marc Strapetz 0220f32e5a Fix possible NegativeArraySizeException in PackIndexV1
Due to an integer overflow bug, the current "Index file is too large
for jgit" check did not work properly and subsequently a
NegativeArraySizeException was raised.

Change-Id: I2736efb28987c29e56bc946563b7fa781898a94a
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2020-08-25 12:42:53 -04:00
Thomas Wolf 2990ad66ad FS: use binary search to determine filesystem timestamp resolution
Previous code used a minimum granularity of 1 microsecond and would
iterate 233 times on a system where the resolution is 1 second (for
instance, Java 8 on Mac APFS).

New code uses a binary search between the maximum we care about (2
seconds) and zero, with a minimum granularity of also 1 microsecond.
This takes at most 19 iterations (guaranteed). For a file system with 1
second resolution, it takes 4 iterations (1s, 0.5s, 0.8s, 0.9s). With
an up-front check at 1 microsecond and at 1 millisecond this performs
equally well as the old code on file systems with a fine resolution.
(For instance, Java 11 on Mac APFS.)

Also handle obscure cases where the file timestamp implementation may
yield bogus values (as observed on HP NonStop). If such an error case
occurs, log a warning and abort the measurement at the last good value.

Bug: 565707
Change-Id: I82a96729b50c284be7c23fbdf3d0df1bddf60e41
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-23 11:39:24 +02:00
Matthias Sohn e9c7ba6fdc Do not prematurely create directory of jgit's XDG config file
LockFile.lock() will create it anyway when the config file is created.

Bug: 565637
Change-Id: I078b89a695193fd76f130f6de7ac1cf26d2f8f0f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-22 22:57:14 +02:00
Thomas Wolf b6ca1993b0 FS: write to JGit config in a background thread
Otherwise locking problems may ensue if the JGit config itself is
on a different file system. Since the internal is already updated,
it is not really important when exactly the value gets persisted.
By queueing up separate Runnables executed by a single thread we
avoid concurrent write access to the JGit config, and nested calls
to getFileStoreAttributes(Path) result in serialized attempts to
write.

The thread for writing the config must not be a daemon thread. If
it were, JVM shutdown might kill it anytime, which may lead to
the config not being written, or worse, a config.lock file being
left behind.

Bug: 566170
Change-Id: I07e3d4c5e029d3cec9ab5895002fc4e0c7948c40
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-21 09:00:06 +02:00
Thomas Wolf eec9b55dcf FS: don't cache fallback if running in background
If the background job is a little late, the true result might
arrive and be cached later. So make sure we don't cache the large
fallback resolution in the per-directory cache. Otherwise we'd work
with the large fallback until the next restart.

Bug: 566170
Change-Id: I7354a6cfddfc0c05144bb0aa41c23029bd4f6af0
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-20 22:17:40 +02:00
Thomas Wolf efd1cc05af Keep line endings for text files committed with CR/LF on text=auto
Git never converts line endings if the version in the repository is a
text file with CR/LF and text=auto. See [1]: "When the file has been
committed with CRLF, no conversion is done."

Because the sentence just before is about converting line endings on
check-in, I had understood that in commit 60cf85a [2] to mean that no
conversion on check-in was to be done. However, as bug 565048 and a
code inspection of the C git code showed it really means no conversion
is done on check-in *or check-out*.

If the text attribute is not set but core.autocrlf = true, this is
the same as text=auto eol=crlf. C git does not convert on check-out
even on text=auto eol=lf if the index version is a text file with
CR/LF.

For check-in, one has to look at the intended target, which is done
in WorkingTreeIterator since commit 60cf85a. For check-out, it can
be done by looking at the source and can thus be done in the
AutoLFOutputStream.

Additionally, provide a constructor for AutoLFInputStream to do
the same; for cases where the equivalent of a check-out is done via
an input stream obtained from a blob. (EGit does that in its
GitBlobStorage for the Eclipse compare framework; it's more efficient
than using a TemporaryBuffer and DirCacheCheckout.getContent(), and
it avoids the need for a temporary file.)

Adapt existing tests, and add new checkout and merge tests to verify
the resulting files have the correct line endings.

EGit's GitBlobStorage will need to call the new version of
EolStreamTypeUtil.wrapInputStream().

[1] https://git-scm.com/docs/gitattributes#Documentation/gitattributes.txt-Settostringvalueauto
[2] https://git.eclipse.org/r/c/jgit/jgit/+/127324

Bug: 565048
Change-Id: If1282ef43e2abd00263541bd10a01fe1f5c619fc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-17 08:52:55 +02:00
Thomas Wolf 71aeedb6ec Delay WindowCache statistics JMX MBean registration
The WindowCache is configured statically with a default
WindowCacheConfig. The default config says (for backwards
compatibility reasons) to publish the MBean. As a result,
the MBean always gets published.

By delaying the MBean registration until the first call to
getInstance() or get(PackFile, long) we can avoid the forced
registration and do it only if not re-configured in the meantime
not to publish the bean. (As is done by Egit, to avoid a very
early costly access to the user and system config during plug-in
activation.)

Bug: 563740
Change-Id: I8a941342c0833acee2107515e64299aada7e0520
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-16 14:41:21 +02:00
Thomas Wolf e9cb0a8e47 DirCache: support index V4
Index format version 4 was introduced in C git in 2012. It's about
time that JGit can deal with it.

Version 4 added prefix path compression. Instead of writing the full
path for each index entry to disk, only the difference to the previous
entry's path is written: a variable-encoded int telling how many bytes
to remove from the previous entry's path to get the common prefix,
followed by the new suffix.

Also, cache entries in a version 4 index are not padded anymore.

Internally, version 3 and version 4 index entries are identical; it's
only the stored format that changes.

Implement this path compression, and make sure we write an index file
that we read previously in the same format. (Only changing from version
2 to version 3 if there are extended flags.)

Add support for the "feature.manyFiles" and the "index.version" git
configs, and honor them when writing a new index file.

Add tests, including a compatibility test that verifies that JGit can
read a version 4 index generated by C git and write an identical
version 4 index.

Bug: 565774
Change-Id: Id83241cf009e50f950eb42f8d56b834fb47da1ed
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-15 12:47:45 +02:00
Thomas Wolf 72b111ecd7 Update javadoc for RemoteSession and SshSessionFactory
The timeout on RemoteSession.exec() cannot be a timeout for the
whole command. It can only be a timeout for setting up the process;
after that it's the application's responsibility to implement some
timeout for the execution of the command, for instance by calling
Process.waitFor(int, TimeUnit) or through other means.

Sessions returned by an SshSessionFactory are already connected and
authenticated -- they must be, because RemoteSession offers no
operations for connecting or authenticating a session.

Change the implementation of SshdExecProcess.waitFor() to wait
indefinitely. The original implementation used the timeout from
RemoteSession.exec() because of that erroneous javadoc.

Change-Id: I3c7ede24ab66d4c81f72d178ce5012d383cd826e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-10 22:51:34 +02:00
Thomas Wolf 24fdc1d039 Fix JSchProcess.waitFor() with time-out
SshSupport.runSshCommand() had a comment that wait with time-out
could not be used because JSchProcess.exitValue() threw the wrong
unchecked exception when the process was still running.

Fix this and make JSchProcess.exitValue() throw the right exception,
then wait with a time-out in SshSupport.

The Apache sshd client's SshdExecProcess has always used the correct
IllegalThreadStateException.

Add tests for SshSupport.runCommand().

Change-Id: Id30893174ae8be3b9a16119674049337b0cf4381
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-08-10 22:51:34 +02:00
Jonathan Nieder 86aa6deff4 FilterSpec: Use BigInteger.ZERO instead of valueOf(0)
This just simplifies a bit by avoiding an unneeded method call.

Change-Id: I6d8d2fc512d8f8a82da73c355017d0abf833a13b
2020-07-31 19:01:35 -07:00
Jonathan Nieder 3c807e0158 Do not send empty blob in response to blob:none filter
If I create a repository containing an empty file and clone it
with

	git clone --no-checkout --filter=blob:none \
		https://url/of/repository

then I would expect no blobs to be transferred over the wire.  Alas,
JGit rewrites filter=blob:none to filter=blob:limit=0, so if the
repository contains an empty file then the empty blob gets
transferred.

Fix it by teaching JGit about filters based on object type to
complement the existing filters based on object size.  This prepares
us for other future filters such as object:none.

In particular, this means we do not need to look up the size of the
filtered blobs, which should speed up clones.  Noticed by Anna
Pologova and Terry Parker.

Change-Id: Id4b234921a190c108d8be2c87f54dcbfa811602a
Signed-off-by: Jonathan Nieder <jrn@google.com>
2020-07-29 21:04:20 -07:00
Jonathan Nieder dceedbcd6e Add support for tree filters when fetching
Teach the FilterSpec serialization code about tree filters so they can
be communicated over the wire and understood by the server.

While we're here, harden the FilterSpec serialization code to throw
IllegalStateException if we encounter a FilterSpec that cannot be
expressed as a "filter" line.  The only public API for creating a
Filterspec is to pass in a "filter" line to be parsed, so these should
not appear in practice.

Change-Id: I9664844059ffbc9c36eb829e2d860f198b9403a0
Signed-off-by: Jonathan Nieder <jrn@google.com>
2020-07-29 20:52:12 -07:00
Thomas Wolf 9fe5406119 FS_POSIX: avoid prompt to install the XCode tools on OS X
OS X ships with a default /usr/bin/git that is just a wrapper that
at run-time delegates to the selected XCode toolchain, and that
prompts the user to install the XCode command line tools if not
already installed.

This is annoying for people who don't want to do so, since they'll
be prompted on each Eclipse start. Also, since on OS X the $PATH for
applications started via the GUI is not the same as the $PATH as set
via the shell profile, just using /usr/bin/git (which will normally
be found when JGit runs inside Eclipse) may give slightly surprising
results if the user has installed a non-Apple git and changed his
$PATH in the shell such that the non-Apple git is used in the shell.
(For instance by placing /usr/local/bin earlier on the path.) Eclipse
and the shell will use different git executables, and thus different
git system configs.

Therefore, try to find git via bash --login -c 'which git' not only
if we couldn't find it on $PATH but also if we found the default git
/usr/bin/git. If that finds some other git, use that. If the bash
approach also finds /usr/bin/git, double check via xcode-select -p
that an XCode git is present. If not, assume there is no git installed,
and work without any system config.

Bug: 564372
Change-Id: Ie9d010ebd9437a491ba5d92b4ffd1860c203f8ca
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-07-26 15:38:48 -04:00
Matthias Sohn 097f01bfb6 Use LinkedBlockingQueue for executor determining filesystem attributes
Using a fixed thread pool with unbounded LinkedBlockingQueue fixes the
RejectedExecutionException thrown if too many threads try to
concurrently determine filesystem attributes.

Comparing that to an alternative implementation using an unbounded
thread pool instead showed similar performance with the reproducer (in
range of 100-1000 threads in reproducer) on my mac:

threads   time

fixed threadpool up to 5 threads with LinkedBlockingQueue of unlimited
queue size
100       1103 ms
200       1602 ms
300       2369 ms
500       4002 ms
1000      11071 ms

unbounded cached threadpool
100       1108 ms
200       1591 ms
300       2299 ms
500       4577 ms
1000      11196 ms

Bug: 564202
Change-Id: I773da7414a1dca8e548349442dca9b56643be946
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-07-24 00:31:03 +02:00
Matthias Sohn 256bc64f2a Update API warning filters
Change-Id: I0fa9c7725ee15da9a932dab0abfb2525d2401149
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-07-19 12:03:14 +02:00
Matthias Sohn fed5b2e6fb Remove unused imports
Change-Id: I7c44e3603df2dd368cb7c0ba0072413b887b6903
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-07-19 12:00:35 +02:00
David Ostrovsky d35f0ffb7c Bazel: Add workspace status command to stamp final artifact
Include implementation version in jgit library. This version is used
by other products that depend on JGit, and built using Bazel and not
consume officially released artifact from Central or Eclipse own Maven
repository.

Most notably, in Gerrit Code Review JGit agent that was previously
reported as "unknown", is now reported as:

  JGit/v5.8.0.202006091008-r-16-g14c43828d

using this change [1].

[1] https://gerrit-review.googlesource.com/c/gerrit/+/272505

Change-Id: Ia50de9ac35b8dbe9e92d8ad7d0d14cd00f057863
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2020-07-17 01:10:15 +02:00
Thomas Wolf 5332723729 DiffFormatter: correctly deal with tracked files in ignored folders
In JGit 5.0, the FileTreeIterator was changed to skip ignored folders
by default. To catch tracked files inside ignored folders, the tree
walk needs to have a DirCacheIterator, and the FileTreeIterator has
to know about that DirCacheIterator via setDirCacheIterator(). (Or
the optimization has to be switched off explicitly via
setWalkIgnoredDirectories(true).)

Skipping ignored directories is an important optimization in some
cases, for instance in node.js/npm projects, where we'd otherwise
traverse the whole huge and deep hierarchy of the typically ignored
node_modules folder.

While all uses of WorkingTreeIterator in JGit had been adapted,
DiffFormatter was forgotten. To make it work correctly (again) also
for such cases, make it set up a WorkingTreeeIterator automatically,
and make sure the WorkingTreeSource can find such files, too. Also
pass the repository to the TreeWalks used inside the DiffFormatter
to pick up the correct attributes, filters, and line-ending settings.

Bug: 565081
Change-Id: Ie88ac81166dc396ba28b83313964c1712b6ca199
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-07-17 00:50:24 +02:00
Matthias Sohn b5ba237559 Prepare 5.8.2-SNAPSHOT builds
Change-Id: I33327417cd5f5b55e860c5d9c6ee06cac7d10b44
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-07-14 20:53:51 +02:00
Matthias Sohn 246954e0d6 JGit v5.8.1.202007141445-r
Change-Id: Ib63831292eded15af18ed30a9653831dadfd0d4a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-07-14 20:43:21 +02:00
Thomas Wolf 9b033a1b6d Fix writing GPG signatures with trailing newline
Make sure we don't produce a spurious empty line at the end.

Bug: 564428
Change-Id: Ib991d93fbd052baca65d32a7842f07f9ddeb8130
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-07-08 09:28:29 +02:00
David Pursehouse 8774f54190 Improve error message when receive.maxCommandBytes is exceeded
The message "Too many commands" implies there is a hard limit on the
number of commands, which isn't the case. The limit is on the total
size of the received data, as explained in change I84317d396 which
introduced the configuration setting receive.maxCommandBytes:

  shorter reference names allow for more commands, longer reference
  names permit fewer commands per batch.

Change the message to:

  Commands size exceeds limit defined in receive.maxCommandBytes

Change-Id: I678b78f919b2fec8f8058f3403f2541c26a5d00e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-06-29 08:57:42 +09:00
Minh Thai 9719ca411e MergedReftable: Include the last reftable in determining minUpdateIndex
MergedReftable ignores the last reftable in the stack while calculating the
minUpdateIndex.

Update the loop indices to include all reftables in the minUpdateIndex
calculation, while skipping position 0 as it is read outside the loop.

Change-Id: I12d3e714581e93d178be79c02408a67ab2bd838e
Signed-off-by: Minh Thai <mthai@google.com>
2020-06-22 17:14:35 -07:00
Yunjie Li b94758441d PackBitmapIndex: Not buffer inflated bitmap during bitmap creation.
Currently we're buffering the inflated bitmap entry in
BasePackBitmapIndex to optimize running time. However, this will use
lots of memory during the creation of the pack bitmap index file.

And change 161456, which rewrote the entire getBitmap method, increased
the fetch latency significantly.

This commit introduces getBitmapWithoutCaching method which is used in
the pack bitmap index file creation only and aims to save memory during
garbage collection and not increase fetch latency.

Change-Id: I7b982c9d4e38f5f6193eaa03894e894ba992b33b
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-06-18 12:36:42 -07:00
Matthias Sohn 13802cd592 Remove trailing whitespace
Change-Id: I1635b79c8051699a3a5e78a4cef8d2014be4e92c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-06-10 10:48:55 +02:00
Matthias Sohn 6abe695aa5 Prepare 5.8.1-SNAPSHOT builds
Change-Id: Ic654fb45abe4e94f4eee532af0f4278d372d37f5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-06-09 16:45:56 +02:00
Matthias Sohn 8e79d5a290 JGit v5.8.0.202006091008-r
Change-Id: I2020e9821c359b90b7c830031945e2fc659ea607
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-06-09 16:07:50 +02:00
Matthias Sohn 855842af19 Prepare 5.9.0-SNAPSHOT builds
Change-Id: Ia998e2772df1285a4c674b07201f15d53156eb78
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-06-09 12:51:12 +02:00
Matthias Sohn 001d747419 Merge "Merge branch 'stable-5.7'" 2020-06-05 16:56:32 -04:00
Jack Wickham 259d2540a3 Add getter for unpackErrorHandler in ReceivePack
The current mechanism for updating the unpack error handler requires
that the error handler is replaced entirely, including communicating
the error to the user. Adding a getter means that delegating
implementations can be constructed so that the error can be processed
before sending to the user, for example for logging.

Change-Id: I4b6f78a041d0f6f5b4076a9a5781565ca3857817
Signed-off-by: Jack Wickham <jwickham@palantir.com>
2020-06-05 15:25:22 -04:00
David Pursehouse 1b4b05d4a3 Merge branch 'stable-5.7'
* stable-5.7:
  ObjectDirectoryInserter: Open FileOutputStream in try-with-resource
  ObjectDirectoryInserter: Remove redundant 'throws' declarations
  ObjectDirectory: Further clean up insertUnpackedObject
  ObjectDirectory: Explicitly handle NoSuchFileException
  ObjectDirectory: Fail immediately when atomic move is not supported

Change-Id: I05186baa517388680fcc6825c940c4c772f26d32
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-06-05 17:09:38 +09:00
David Pursehouse c0c7f445f4 ObjectDirectoryInserter: Open FileOutputStream in try-with-resource
Change-Id: Icc569aeefdc79baee5dfb71fb34d881c561dcf52
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-06-05 14:59:54 +09:00
David Pursehouse f4f5d448b6 ObjectDirectoryInserter: Remove redundant 'throws' declarations
ObjectWritingException and FileNotFoundException are subclasses
of IOException, which is already declared. Error does not need
to be explicitly declared.

Change-Id: I879820a33e10ec3a7ef676adc9c9148d2b3c4b27
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-06-05 14:48:15 +09:00
David Pursehouse dac6801b47 ObjectDirectory: Further clean up insertUnpackedObject
- The code to move the file is repeated. Split it out into a
  utility method.

- Remove the catch block for AtomicMoveNotSupportedException which
  is redundant because it's handled in exactly the same way as the
  IOException further down. The only exception we need to explicitly
  handle differently in this block is NoSuchFileException.

- Improve the comments.

Change-Id: Ifc5490953ffb25ecd1c48a06289eccb3f19910c6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-06-05 11:40:08 +09:00
Matthias Sohn aea95b819a Add Git#shutdown for releasing resources held by JGit process
The shutdown method releases
- ThreadLocal held by NLS
- GlobalBundleCache used by NLS
- Executor held by WorkQueue

Bug: 437855
Bug: 550529
Change-Id: Icfdccd63668ca90c730ee47a52a17dbd58695ada
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-06-04 23:51:59 +02:00
Thomas Wolf ed481f96b8 ApplyCommand: use context lines to determine hunk location
If a hunk does not apply at the position stated in the hunk header
try to determine its position using the old lines (context and
deleted lines).

This is still a far cry from a full git apply: it doesn't do binary
patches, it doesn't handle git's whitespace options, and it's perhaps
not the fastest on big patches. C git hashes the lines and uses these
hashes to speed up matching hunks (and to do its whitespace magic).

Bug: 562348
Change-Id: Id0796bba059d84e648769d5896f497fde0b787dd
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-06-04 22:16:12 +02:00
David Ostrovsky 0e87f70e0e Fix ProtectedMembersInFinalClass warning flagged by error prone
Running recent error prone version complaining on that code:

CharacterHead.java:22: error: [ProtectedMembersInFinalClass] Make
members of final classes package-private: <init>
	protected CharacterHead(char expectedCharacter) {
	          ^
    (see https://errorprone.info/bugpattern/ProtectedMembersInFinalClass)
  Did you mean 'CharacterHead(char expectedCharacter) {'

Bug: 562756
Change-Id: Ic46a0b07e46235592f6e63db631f583303420b73
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2020-06-04 16:24:16 +02:00
David Pursehouse 949ee670c6 ObjectDirectory: Explicitly handle NoSuchFileException
On the first attempt to move the temp file, NoSuchFileException can
be raised if the destination folder does not exist. Instead of handling
this implicitly in the catch of IOException and then continuing to
create the destination folder and try again, explicitly catch it and
create the destination folder. If any other IOException occurs, treat
it as an unexpected error and return FAILURE.

Subsequently, on the second attempt to move the temp file, if ANY kind
of IOException occurs, also consider this an unexpected error and
return FAILURE.

In both catch blocks for IOException, add logging at ERROR level.

Change-Id: I9de9ee3d2b368be36e02ee1c0daf8e844f7e46c8
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-06-04 14:07:34 +09:00
David Pursehouse c2ab332e81 ObjectDirectory: Fail immediately when atomic move is not supported
If atomic move is not supported, AtomicMoveNotSupportedException will
be thrown on the first attempt to move the temp file. There is no
point attempting the move operation a second time because it will only
fail for the same reason.

Add an immediate return of FAILURE on the first occasion. Remove the
unnecessary handling of the exception in the second block.

Change-Id: I4658a8b37cfec2d7ef0217c8346e512968d0964c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-06-04 14:06:21 +09:00
David Ostrovsky 7861f82029 Fix InvalidInlineTag error flagged by error prone
Running recent error prone version complaining on that code:

RefDatabase.java:444: error: [InvalidInlineTag] Tag name `linkObjectId`
is unknown.
	 * Includes peeled {@linkObjectId}s. This is the inverse lookup of
	                   ^
    (see https://errorprone.info/bugpattern/InvalidInlineTag)

Bug: 562756
Change-Id: If91da51d5138fb753c0550eeeb9e3883a394123d
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2020-06-01 22:51:37 -04:00
Matthias Sohn 8d2d683655 Decouple JSch from JGit Core
Motivation: JSch serves as 'default' implementations of the SSH
transport. If a client application does not use it then there is no need
to pull in this dependency.

Move the classes depending on JSch to an OSGi fragment extending the
org.eclipse.jgit bundle and keep them in the same package as before
since moving them to another package would break API. Defer moving them
to a separate package to the next major release.

Add a new feature org.eclipse.jgit.ssh.jsch feature to enable
installation. With that users can now decide which of the ssh client
integrations (JCraft JSch or Apache Mina SSHD) they want to install.
We will remove the JCraft JSch integration in a later step due to the
reasons discussed in bug 520927.

Bug: 553625
Change-Id: I5979c8a9dbbe878a2e8ac0fbfde7230059d74dc2
Also-by: Michael Dardis <git@md-5.net>
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2020-06-01 01:46:59 +02:00
Matthias Sohn 77848d635b Decouple BouncyCastle from JGit Core
Motivation: BouncyCastle serves as 'default' implementation of
the GPG Signer. If a client application does not use it there is no need
to pull in this dependency, especially since BouncyCastle is a large
library.

Move the classes depending on BouncyCastle to an OSGi fragment extending
the org.eclipse.jgit bundle. They are moved to a distinct internal
package in order to avoid split packages. This doesn't break public API
since these classes were already in an internal package before this
change.

Add a new feature org.eclipse.jgit.gpg.bc to enable installation. With
that users can now decide if they want to install it.

Attempts to sign a commit if org.eclipse.jgit.gpg.bc isn't available
will result in ServiceUnavailableException being thrown.

Bug: 559106
Change-Id: I42fd6c00002e17aa9a7be96ae434b538ea86ccf8
Also-by: Michael Dardis <git@md-5.net>
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2020-06-01 01:26:22 +02:00
Thomas Wolf 0b2d41b858 Verify that the user home directory is valid
If the determination of the user home directory produces a Java File
object with an invalid path, spurious exceptions may occur at the
most inopportune moments anytime later. In the case in the linked bug
report, start-up of EGit failed, leading to numerous user-visible
problems in Eclipse.

So validate the return value of FS.userHomeImpl(). If converting that
File to a Path throws an exception, log the problem and fall back to
Java system property user.home. If that also is not valid, use null.

(A null user home directory is allowed by FS, and calling in Java
new File(null, "some_string") is fine and produces a File relative
to the current working directory.)

Bug: 563739
Change-Id: If9eec0f9a31a45bd815231706285c71b09f8cf56
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-31 12:47:21 -04:00
Thomas Wolf 089eacb273 WindowCache: conditional JMX setup
Make it possible to programmatically suppress the JMX bean
registration. In EGit it is not needed but can be rather costly
because it occurs during plug-in activation and accesses the
git user config.

Bug: 563740
Change-Id: I07ef7ae2f0208d177d2a03862846a8efe0191956
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-29 23:05:46 +02:00
Christian Halstrick c6213ad33a Merge "RawTextComparator.WS_IGNORE_CHANGE must not compare whitespace" 2020-05-28 08:07:02 -04:00
Thomas Wolf 6f17f9ed3f RawTextComparator.WS_IGNORE_CHANGE must not compare whitespace
Only the presence or absence of whitespace is significant; but not the
actual whitespace characters. Don't compare whitespace bytes.

Compare the C git implementation at [1].

[1] https://github.com/git/git/blob/0d0e1e8/xdiff/xutils.c#L173

Bug: 563570
Change-Id: I2d0522b637ba6b5c8b911b3376a9df5daa9d4c27
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-28 12:06:57 +02:00
Yunjie Li 06a90fdf2e Revert "PackBitmapIndex: Not buffer inflated bitmap in BasePackBitmapIndex"
This reverts commit 3aee92478c, which
increased fetch latency significantly.

Change-Id: Id31a94dff83bf7ab2121718ead819bd08306a0b6
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-27 10:31:54 -07:00
Thomas Wolf 3a499606b1 Builder API to configure SshdSessionFactories
A builder API provides a more convenient way to define a customized
SshdSessionFactory by hiding the subclassing.

Also provide a new interface SshConfigStore to abstract away the
specifics of reading a ssh config file, and provide a way to customize
the concrete ssh config implementation to be used. This facilitates
using an alternate ssh config implementation that may or may not be
based on files.

Change-Id: Ib9038e8ff2a4eb3a9ce7b3554d1450befec8e1e1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-23 16:46:22 +02:00
Thomas Wolf bdb7357228 TransportHttp: abort on time-out or on SocketException
Avoid trying other authentication methods on SocketException or on
InterruptedIOException. SocketException is rather fatal, such as
nothing listening on the peer's port, connection reset, or it could
be a connection time-out.

Time-outs enforced by Timeout{Input,Output}Stream may result in
InterruptedIOException being thrown.

In both cases, it makes no sense to try other authentication methods,
and doing so may wrongly report "authentication not supported" or
"cannot open git-upload-pack" or some such instead of reporting a
time-out.

Bug: 563138
Change-Id: I0191b1e784c2471035e550205abd06ec9934fd00
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-23 11:06:10 +02:00
Thomas Wolf 3dbd1f2fe7 Ignore core.eol if core.autocrlf=input
Config core.eol is to be ignored if core.autocrlf is true or input.[1]
JGit didn't do so when core.autocrlf=input was set.

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-coreeol

Bug: 561877
Change-Id: I5e62e0510d160b5113c1090319af09c2bc1bcb59
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-22 17:09:23 -04:00
Thomas Wolf 3c34e0acbf Attributes: fix handling of text=auto in combination with eol
In Git 2.10.0 the interpretation of gitattributes changed or was fixed
such that "* text=auto eol=crlf" would indeed still do auto-detection
of text vs. binary content.[1] Previously this was identical to
"* text eol=crlf", i.e., treating all files as text.

JGit still did the latter, which caused surprises because it changed
binary files.

[1] https://github.com/git/git/blob/master/Documentation/RelNotes/2.10.0.txt#L248

Bug: 561341
Change-Id: I5b6fb97b5e86fd950a98537b6b8574f768ae30e5
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-22 17:08:52 -04:00
Thomas Wolf 4d7a16257f Include full IssuerFingerprint in GPG signature
Update dependency to Bouncy Castle to 1.65.

Add the IssuerFingerprint as a hashed sub-packet in the signature. If
added unhashed, GPG ignores it.

Bug: 553206
Change-Id: I6807e8e2385e6ec5790f388e4753a44aa9474ebb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-18 23:25:58 +02:00
Matthias Sohn d0f010dd26 Suppress API error for new method BitmapIndex.Bitmap#retrieveCompressed
OSGi semantic versioning allows breaking implementers in a minor
release.

Change-Id: Ib55dc43dd3b50b0ef39a7094190f230210aee4b6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-17 23:11:31 +02:00
Matthias Sohn 91188a7d82 Fix wrong @since tags added in dcb0265
This change was introduced in 5.8.

Change-Id: Ic74ebff5a0547bb55e0401b38f73ebc6e67cace9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-17 23:08:27 +02:00
Terry Parker 55b0203c31 Merge changes I39783eee,I874503ec,Ic942a8e4,I6ec2c3e8,I62cb5030, ...
* changes:
  PackBitmapIndex: Set distance threshold
  PackBitmapIndex: Not buffer inflated bitmap in BasePackBitmapIndex
  PackBitmapIndex: Remove convertedBitmaps in the Remapper
  PackBitmapIndex: Reduce memory usage in GC
  PackBitmapIndex: Add AddToBitmapWithCacheFilter class
  PackBitmapIndex: Add util methods and builder to BitmapCommit
  PackBitmapIndex: Move BitmapCommit to a top-level class
  Refactor: Make retriveCompressed an method of the Bitmap class
2020-05-13 16:34:23 -04:00
Yunjie Li 913234e2ec PackBitmapIndex: Set distance threshold
Setting the distance threshold to 2000 in PackWriterBitmapPreparer to
reduce memory usage in garbage collection. When the threshold is 0, GC
for the msm repository would use about 37 GB memory to complete. After
setting it to 2000, GC can finish in 75 min with about 10 GB memory.

Change-Id: I39783eeecbae58261c883735499e61ee1cac75fe
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li 3aee92478c PackBitmapIndex: Not buffer inflated bitmap in BasePackBitmapIndex
Currently we're buffering the inflated bitmap entry in BasePackBitmapIndex
to optimize running time. However, this will use lots of memory during
the construction of the pack bitmap index file which may cause failure of
garbage collection.

The running time didn't increase significantly, if there's any increase,
after removing the buffering here. The report about usage of time/memory
will come in the next commit.

Change-Id: I874503ecc85714acab7ca62a6a7968c2dc0b56b3
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li e250482c7a PackBitmapIndex: Remove convertedBitmaps in the Remapper
The convertedBitmaps serves for time-optimization purpose. But it's
actually not saving time much but using lots of memory. So remove the
field here to save memory.

Currently the remapper class is only used in the construction of the
bitmap index file. And during the preparation of the file, we're only
getting bitmaps from the remapper when finding objects accessible from
a commit, so bitmap associated with each commit will only be fetched once
and thus the convertedBitmaps would hardly be read, which means that it's
not saving time.

Change-Id: Ic942a8e485135fb177ec21d09282d08ca6646fdb
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li dcb0265436 PackBitmapIndex: Reduce memory usage in GC
Currently, the garbage collection is consistently failing for some large
repositories in the building bitmap phase, e.g.Linux-MSM project:
https://source.codeaurora.org/quic/la/kernel/msm-3.18

Historically, bitmap index creation happened in 3 phases:
1. Select the commits to which bitmaps should be attached.
2. Create all bitmaps for these commits, stored in uncompressed format
in the PackBitmapIndexBuilder.
3. Deltify the bitmaps and write them to disk.

We investigated the process. For phase 2 it's most efficient to create
bitmaps starting with oldest commit and moving to the newest commit,
because the newer commits are able to reuse the work for the old ones.
But for bitmap deltification in phase 3, it's better when a newer
commit's bitmap is the base, and the current disk format writes bitmaps
out for the newest commits first.

This change introduces a new collection to hold the deltified and
compressed representations of the bitmaps, keeping a smaller subset of
commits in the PackBitmapIndexBuilder to help make the bitmap index
creation more memory efficient.

And in this commit, we're setting DISTANCE_THRESHOLD to 0 in the
PackWriterBitmapPreparer, which means the garbage collection will not
have much behavoir change and will still use as much memory as before.

Change-Id: I6ec2c3e8dde11805af47874d67d33cf1ef83660e
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li 067d946090 PackBitmapIndex: Add AddToBitmapWithCacheFilter class
Add a new revwalk filter, AddToBitmapWithCachedFilter. This filter updates
a client-provided {@code BitmapBuilder} as a side effect of a revwalk.
Similar to {@code AddToBitmapFilter}, it short circuits the walk when it
encounters a commit which is included in the provided bitmap's BitmapIndex.
It also short circuits the walk if it encounters the client-provided
cached commit.

Change-Id: I62cb503016f4d3995d648d92b82baab7f93549a9
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li b1d4b45708 PackBitmapIndex: Add util methods and builder to BitmapCommit
Add some utility methods and a builder class for BitmapCommit class in
preparation for improving the memory footprint of GC's bitmap generation
phase.

Change-Id: Ice3d257fc26f3917a65a64eaf53b508b89043caa
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li d23254ee57 PackBitmapIndex: Move BitmapCommit to a top-level class
Move BitmapCommit from inside the PackWriterBitmapPreparer to a new
top-level class in preparation for improving the memory footprint of GC's
bitmap generation phase.

Change-Id: I4d404a5b3a34998b441d23105197f33d32d39670
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:15 -07:00
Yunjie Li 840e414d0b Refactor: Make retriveCompressed an method of the Bitmap class
Make retrieveCompressed() a method of Bitmap interface to avoid type
casting and later reuse in improving the memory footprint of GC's bitmap
generation phase.

Change-Id: I098d85105cf17af845d43b8c71b4ca48b02fd7da
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-05-12 17:32:05 -07:00
Matthias Sohn a379d007db Fix downloading LFS Object fails behind proxy
When downloading LFS objects also accept response code 203 as successful
download. This response may be seen when downloading via a proxy.

Bug: 563022
Change-Id: Iee85fdb451b33369d08859872e5bfc2a67dffa6d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-12 10:26:45 -04:00
Terry Parker 0642e49f97 Merge changes I6543c2e1,I21ed029d
* changes:
  ReceivePack: adding IterativeConnectivityChecker
  Moving transport/internal -> internal/transport
2020-05-11 17:40:20 -04:00
Demetr Starshov 9075beefb1 ReceivePack: adding IterativeConnectivityChecker
Introduce an IterativeConnectivityChecker which runs a connectivity
check with a filtered set of references, and falls back to using the
full set of advertised references.

It uses references during first check attempt:
- References that are ancestors of an incoming commits (e.g., pushing
a commit onto an existing branch or pushing a new branch based on
another branch)
- Additional list of references we know client can be interested in
(e.g. list of open changes for Gerrit)

We tested it inside Google and it improves connectivity for certain
topologies. For example connectivity counts for
chromium.googlesource.com/chromium/src:

percentile_50: 1923 (was: 22777)
percentile_90: 23272 (was: 353003)
percentile_99: 345522 (was: 353435)

This saved ~2 seconds on every push to this repository.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I6543c2e10ed04622ca795b195665133e690d3b10
2020-05-08 17:57:20 -07:00
Demetr Starshov 519cb1e91b Moving transport/internal -> internal/transport
Moving transport related internal classes into dedicated subpackage in
o/e/j/internal package.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I21ed029d359f5f7d8298f102efbb4b1dcdf404ad
2020-05-08 17:57:20 -07:00
Nail Samatov f30e48914e Fix error occurring during checkout
Fix NullPointerException occurring when calling
CheckoutCommand with forced == true option when
the branch isn't changed and there is deleted
uncommitted file.

Change-Id: I99bf1fc25e6889f07092320d7bc2772ec5d341b5
Signed-off-by: Nail Samatov <sanail@yandex.ru>
2020-05-07 12:18:44 +03:00
Matthias Sohn 231c44d553 ApplyCommand: use Files#copy to copy file
This should be faster.

Change-Id: I404ec5e66731b3cf7a8e621cf1ff8748d109ea69
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-05-06 10:16:35 +02:00
Jack Wickham b6d70a66ae Apply hunks when renaming or copying from patch files
When applying a patch that contains renames or copies using ApplyCommand,
also apply all hunks that apply to the renamed or copied file.

Change-Id: I9f3fa4370458bd7c14beeb2e2b49e846d70203cb
Signed-off-by: Jack Wickham <jwickham@palantir.com>
2020-05-06 08:50:08 +02:00
Jack Wickham d69c0ef5bd Create parent directories when renaming a file in ApplyCommand
Before this change, applying a patch will fail if the destination directory
doesn't exist; after, the necessary parent directories are created.

If renaming the file fails, the directories won't be deleted, so this change
isn't atomic. However, ApplyCommand is already not atomic - if one hunk fails
to apply, other hunks still get applied - so I don't think that is a blocker.

Change-Id: Iea36138b806d4e7012176615bcc673756a82f365
Signed-off-by: Jack Wickham <jwickham@palantir.com>
2020-05-06 08:50:08 +02:00
Ivan Frade 0a2a094fea Reduce BitmappedObjectReachabilityChecker visibility
ObjectReachabilityChecker interface is the only public API. The
implementation is instantiated by ObjectWalk and doesn't need to be
visible outside the package.

Change-Id: I5b97bb98990cded637686bdc15c9655330b7780f
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-05-04 11:45:15 -07:00
Thomas Wolf 9f07528ff8 Add missing @since to new API
Change-Id: Ie7c3481aba515d9c9b0a152db17b5a9dc74e7ede
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-05-04 09:30:24 +02:00
Ivan Frade 70ec9a83b8 ObjectReachabilityCheckers: Make walk member final
It is only assigned on initialization.

Change-Id: I36f68b3f511236e66cc7a4f56ba49252b317276a
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-30 10:23:29 -07:00
Ivan Frade c9d7285e80 UploadPack: Use more relevant refs first in object reachability check
The bitmap-bassed object reachability checker, tries to find the objects
in the first starter, then adding the second starter... and so on. This
rewards passing the most popular refs first.

Order the refs with heads first, then tags, then others (e.g. changes)
for the object reachability checker. Using streams, delay also the
resolution of the ref to RevObject until necessary.

Change-Id: I9414b76754d7c0ffee1e2eeed6939895c8e92cbe
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade a661e2e9eb UploadPack: Refactor to generalize the object reachability checks
ObjectWalk#createObjectReachabilityChecker() returns the best
implementation for the repo. UploadPack can use the interface and fold
the with/without commits cases in one code path.

Change-Id: I857c11735d1d8e36c3ed8185ff11de8a62e86540
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade 6bc04bdc02 UploadPack: Use BitmappedReachabilityChecker for not advertised wants
Change-Id: Ifea971d5c0309e28a909441ee8a6f1e62397d6d3
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:39 -07:00
Ivan Frade 003002c1cb revwalk: Introduce bitmap-based object reachability checker
Change-Id: I0b1a2bd21f98894862aab339f8c2e4a417897b89
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 22:44:33 -07:00
Ivan Frade 20bb312421 revwalk: Extract ObjectReachabilityChecker interface
Extract ObjectReachabilityChecker interface from the walk-based
implementation, to add a bitmapped based implementation later.

Refactor the test case to use it for both implementations.

Change-Id: Iaac7c6b037723811956ac22625f27d3b4d742139
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 16:56:32 -07:00
Ivan Frade ae26fa19b7 UploadPack: Extract walk-based reachability check
Preparing the code to optimize the bitmap-based object reachability
checker.  We are mirroring first the commit reachability checker
structure (interface + 2 implementations).

Move the walk-base reachability checker to its own class.

This class is public at the moment. Later ObjectWalk will return an
interface and this implementation will be package-private.

Change-Id: Ifac70094e1af137291c3607d95e689992f814b26
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-04-28 16:40:52 -07:00
Matthias Sohn c845db3ec2 RefTreeBatch: fix unclosed resource warning
Change-Id: I7d630d11d3ef2ff2b74ec7a7018760b7b4931ad2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:48:14 +02:00
Matthias Sohn 7ae3507998 CherryPickCommand: fix unclosed resource warning
Change-Id: I2dc76efd16d887048a5cb4af2b2d8e6faa4f22cd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:48:14 +02:00
Matthias Sohn 560ef875e3 URIish: suppress non-localized message warning
Change-Id: I3ec37c67ba6f00ad8bf396aa3261dd90f35789ea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-04-28 08:21:06 +09:00
Minh Thai d9f84b0b7c UploadPack: Clear advertised ref map after negotiation
After negotiation phase of a fetch, the advertised ref map is no longer used and
can be safely cleared. For >1GiB repos object selection and packfile writing may
take 10s of minutes. For the chromium.googlesource.com/chromium/src repo, this
advertised ref map is >400MiB. Returning this memory to the Java heap is a major
scalability win.

Change-Id: I00d453c5ef47630c21f199e333e1cfcf47b7e92a
Signed-off-by: Minh Thai <mthai@google.com>
2020-04-23 12:14:02 -07:00
Konrad Windszus 54a2d48008 Fix human name for local .bundle files
Bug: 560903
Change-Id: I15d45330398cc573940265d16a2db29ddce085aa
Signed-off-by: Konrad Windszus <konrad_w@gmx.de>
2020-04-21 07:46:50 +02:00
Matthias Sohn a0aebbd528 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  Remove double blank from sentence start
  Bump Bazel version to 3.0.0

Change-Id: I26c3a8345020239d1c2ec5c6f70a633b43ddab86
2020-04-18 00:30:15 +02:00
Michael Keppler a79c5b1f10 Remove double blank from sentence start
Multiple whitespaces are not normalized when reading properties files,
therefore leading to unwanted space/indentation in console or UI output.

Change-Id: I1f5224fe359e0cac493e0237872afc75dc8b9fbe
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ebbc3efce7)
2020-04-17 14:36:50 +09:00
Michael Keppler ebbc3efce7 Remove double blank from sentence start
Multiple whitespaces are not normalized when reading properties files,
therefore leading to unwanted space/indentation in console or UI output.

Change-Id: I1f5224fe359e0cac493e0237872afc75dc8b9fbe
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
2020-04-14 09:31:50 +02:00
Terry Parker 9c67f680c8 Merge "ReceivePack: Use error message if set" 2020-04-06 11:20:46 -04:00
Minh Thai 0eeed6ad2b Scan through all merged reftables for max/min update indices
Since reftables might have update index ranges that are overlapped.

Change-Id: I8f8215b99a0a978d4dd0155dbaf33e5e06ea8202
Signed-off-by: Minh Thai <mthai@google.com>
(cherry picked from commit 06748c205c)
2020-04-05 17:26:20 -04:00
Alexander Nittka bc4ed530a5 FileUtils: improve delete (Windows)
Ensure files are writable before trying to delete them.

Bug: 408846
Change-Id: I930a547594bba853c33634ae54bd64d236afade3
Signed-off-by: Alexander Nittka <alex@nittka.de>
2020-04-03 19:30:31 +02:00
Thomas Wolf 9aaa58052b Merge branch 'stable-5.7'
* stable-5.7:
  FS.runInShell(): handle quoted filters and hooksPath containing blanks
  Handle non-normalized index also for executable files
  Prepare 5.7.1-SNAPSHOT builds
  JGit v5.7.0.202003110725-r

Change-Id: I8a8580e44bfa05989d476cf22a029abd4fd407c6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-04-03 10:40:30 +02:00
Thomas Wolf 3ae0b5cfeb Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  FS.runInShell(): handle quoted filters and hooksPath containing blanks
  Handle non-normalized index also for executable files

Change-Id: I240377e87c073ee7a621a88e39fc319c59fa037a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-04-03 10:21:00 +02:00
Thomas Wolf 2640d38f14 FS.runInShell(): handle quoted filters and hooksPath containing blanks
Revert commit 2323d7a. Using $0 in the shell command call results in
the command string being taken literally. That was introduced to fix
a problem with backslashes, but is actually not correct.

First, the problem with backslashes occurred only on Win32/Cygwin,
and has been properly fixed in commit 6f268f8.

Second, this is used only for hooks (which don't have backslashes in
their names) and filter commands from the git config, where the user
is responsible for properly quoting or escaping such that the commands
work.

Third, using $0 actually breaks correctly quoted filter commands
like in the bug report. The shell really takes the command literally,
and then doesn't find the command because of quotes.

So revert this change.

At the same time there's a related problem with hooks. If the path to
the hook contains blanks, runInShell() would also fail to find the
hook. In this case, the command doesn't come from user input but is
just a Java File object with an absolute path containing blanks. (Can
occur if core.hooksPath points to such a path with blanks, or if the
repository has such a path.)

The path to the hook as obtained from the file system must be quoted.

Add a test for a hook path with a blank.

This reverts commit 2323d7a1ef.

Bug: 561666
Change-Id: I4d7df13e6c9b245fe1706e191e4316685a8a9d59
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-04-03 08:18:39 +02:00
Matthias Sohn 26b23b2713 Define constants for pack config option keys
Change-Id: Ifb8227cb62370029d6774f2a22b15d6478c713ca
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-04-03 11:49:25 +09:00
Karsten Thoms bd42c92ab3 Fix javadoc typo
Bug: 499934
Change-Id: I0d6ee56fad4472972f69cc1e7cb2a727e95ab6b4
Signed-off-by: Karsten Thoms <karsten.thoms@karakun.com>
2020-04-02 10:54:43 +02:00
Masaya Suzuki 7ad42d7416 ReceivePack: Use error message if set
ReceiveCommand can have an error message. This is shown only for some
cases even if it's set. This change uses the error message if it's set,
and fallback to the default message if unset.

Change-Id: I8d906e71ad08cf49bcdb28caea8fcc66798c68ff
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2020-03-27 12:23:36 -07:00
Thomas Wolf 24f82b533a Handle non-normalized index also for executable files
Commit 60cf85a4 corrected the handling of check-in for files where
the index version is non-normalized, i.e., contains CR-LF line endings.
However, it did so only for regular files, not executable files.

Bug: 561438
Change-Id: I372cc990c5efeb00315460f36459c0652d5d1e77
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-03-25 09:50:12 +01:00
Minh Thai 06748c205c Scan through all merged reftables for max/min update indices
Since reftables might have update index ranges that are overlapped.

Change-Id: I8f8215b99a0a978d4dd0155dbaf33e5e06ea8202
Signed-off-by: Minh Thai <mthai@google.com>
2020-03-20 11:12:34 -07:00
Ivan Frade 7d3b6308fc ResolveMerger: Ignore merge conflicts if asked so
The recursive merge strategy builds a virtual ancestor merging
recursively the common bases (when more than one) between the
want-to-merge commits. While building this virtual ancestor, content
conflicts are ignored, but current code doesn't do so when a file is
removed.

This was spotted in [1], for example. Merging two commits to build the
virtual ancestor bumped into a conflict (modified in one side, deleted
in the other) that stopped the process.

Follow the "spec" and in case of conflict leave the unmerged content in
the index and working trees.

[1] https://android-review.googlesource.com/c/kernel/common/+/1228962

Change-Id: Ife9c32ae3ac3a87d3660fa1242e07854b65169d5
Signed-off-by: Ivan Frade <ifrade@google.com>
2020-03-19 16:36:21 -07:00
Christian Halstrick d22a8a2510 Merge "tag option for clone command" 2020-03-15 18:11:14 -04:00
Alexander Nittka 769606d059 tag option for clone command
Allow explicitly setting the tag option for the remote configuration
when cloning a repository.

Bug: 561021
Change-Id: Iac43268a2bb231ae7599c3255bf555883d34fa32
Signed-off-by: Alexander Nittka <alex@nittka.de>
2020-03-15 15:34:21 +01:00
Alex Spradlin cf8c84c6d1 RevWalk: fix bad topo flags error message
The error message for an Exception thrown by StartGenerator when given
both the TOPO flag and the TOPO_KEEP_BRANCH_TOGETHER flag mentions a
non-existent flag, TOPO_NON_INTERMIX. The error message was introduced
in commit e498d43.

Replace TOPO_NON_INTERMIX with TOPO_KEEP_BRANCH_TOGETHER in the error
message of an Exception thrown by the StartGenerator when the TOPO flag
is provided together with the TOPO_KEEP_BRANCH_TOGETHER flag.

Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
Change-Id: Id24640dc08e96a196508fe38ce144aa7e035082f
2020-03-12 09:04:36 -07:00
Alex Spradlin e498d43186 RevWalk: new topo sort to not mix lines of history
The topological sort algorithm in TopoSortGenerator for RevWalk may mix
multiple lines of history, producing results that differ from C git's
git-log whose man page states: "Show no parents before all of its
children are shown, and avoid showing commits on multiple lines of
history intermixed." Lines of history are mixed because
TopoSortGenerator merely delays producing a commit until all of its
children have been produced; it does not immediately produce a commit
after its last child has been produced.

Therefore, add a new RevSort option called TOPO_KEEP_BRANCH_TOGETHER
with a new topo sort algorithm in TopoNonIntermixGenerator. In the
Generator, when the last child of a commit has been produced, unpop
that commit so that it will be returned upon the subsequent call to
next(). To avoid producing duplicates, mark commits that have not yet
been produced as TOPO_QUEUED so that when a commit is popped, it is
produced if and only if TOPO_QUEUED is set.

To support nesting with other generators that may produce the same
commit multiple times like DepthGenerator (for example, StartGenerator
does this), do not increment parent inDegree for the same child commit
more than once.

Commit b5e764abd2 modified the existing
TopoSortGenerator to avoid mixing lines of history, but it was reverted
in e40c38ab08 because the new behavior
caused problems for EGit users. This motivated adding a new Generator
for the new behavior.

Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
Change-Id: Icbb24eac98c00e45c175b01e1c8122554f617933
2020-03-11 15:39:38 -07:00
Matthias Sohn 0d78edad53 Prepare 5.7.1-SNAPSHOT builds
Change-Id: I29cd1b9872603810f9c4d42424ac326fedf5569f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-11 13:35:23 +01:00
Matthias Sohn 0713f6e962 JGit v5.7.0.202003110725-r
Change-Id: Ia4a9122eeb4eea0f379f6611984cfe1143a32f8d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-11 13:24:54 +01:00
Thomas Wolf 04e16afb05 TransportHttp: support HTTP response 308 Permanent Redirect
RFC 7538[1] added HTTP response code 308, signifying a permanent
redirect that, contrary to the older 301, does not allow changing
the request method from POST to GET.

[1] https://tools.ietf.org/html/rfc7538

Bug: 560936
Change-Id: Ib65f3a3ed75db51d74d1fe81d4abe6fe92b0ca12
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-03-10 11:34:48 +01:00
Thomas Wolf ac83e471d8 Remove unused API problem filters
Change-Id: I137cddaf0aecdf9f4bd1427fa9c1af1af4f5b546
2020-03-10 11:11:14 +01:00
Matthias Sohn d408e41e4e Prepare 5.7.1-SNAPSHOT builds
Change-Id: I23748c20b69a1cfb47c5cd0d7858182d376b3ce1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-09 23:37:43 +01:00
Matthias Sohn 8fab712f75 Prepare 5.8.0-SNAPSHOT builds
Change-Id: I056b45806a82eae80177932e42e3dc806015351a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-09 22:47:10 +01:00
Matthias Sohn f5793bea53 JGit v5.7.0.202003090808-r
Change-Id: Ia23bffa3952687f6107d496a8932d54b11c12412
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-09 14:09:24 +01:00
Matthias Sohn fb0858e9c9 Merge branch 'stable-5.6'
* stable-5.6:
  Silence API errors introduced by 093fbbd1
  Bump Bazel version to 2.2.0
  Expose FileStoreAttributes.setBackground()
  Update reftable storage repo layout

Change-Id: I237eaaed7991e8bbd56a7624f47bbba985330026
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-05 15:44:42 +01:00
Matthias Sohn 2415571370 Silence API errors introduced by 093fbbd1
Change-Id: I1c9d5a25bd06a1152e953c45b683375cb05aa254
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-05 13:53:10 +01:00
Michael Dardis d7f304045a Add validation to hex decoder
Does not fix any issue but prevents user from shooting themselves in the
foot with improper configuration.

Suggested by Demetr Starshov at https://git.eclipse.org/r/#/c/157681/

Change-Id: I006d65022f0a7d4066970825d00080c59404fdc3
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-04 16:34:01 +01:00
Alex Blewitt 093fbbd11e Expose FileStoreAttributes.setBackground()
The FS.setAsyncFileStoreAttributes() static method calls
FileStoreAttributes.setBackground() as its implementation, but there are
other public attributes on this inner class already and there isn't a
real reason why this needs to be private.

By making it public we allow callers to be able to invoke it directly.
Although it doesn't appear that it would make a difference, by calling a
static method on the FS class, all static fields and the transitive
closure of class dependencies must be loaded and initialised, which can
be non-trivial.

Callers referring to FS.setAsyncFileStoreAttributes() may be replaced
with FS.FileStoreAttributes.setBackground() with no change of behaviour
other than improved performance due to less class loading required.

Bug: 560527
Change-Id: I9538acc90da8d18f53fd60d74eb54496857f93a5
Signed-off-by: Alex Blewitt <alex.blewitt@gmail.com>
2020-03-04 15:59:53 +01:00
Han-Wen Nienhuys 30fb4808f2 Update reftable storage repo layout
The change Ic0b974fa (c217d33, "Documentation/technical/reftable:
improve repo layout") defines a new repository layout, which was
agreed with the git-core mailing list.

It addresses the following problems:

* old git clients will not recognize reftable-based repositories, and
  look at encompassing directories.

* Poorly written tools might write directly into
  .git/refs/heads/BRANCH.

Since we consider JGit reftable as experimental (git-core doesn't
support it yet), we have no backward compatibility. If you created a
repository with reftable between mid-Nov 2019 and now, you can do the
following to convert:

  mv .git/refs .git/reftable/tables.list
  git config core.repositoryformatversion 1
  git config extensions.refStorage reftable

Change-Id: I80df35b9d22a8ab893dcbe9fbd051d924788d6a5
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-03-04 15:45:01 +01:00
Jonathan Nieder fb45137a7c Merge "ObjectWalk: Add null check before skip tree." 2020-03-02 13:37:59 -05:00
Matthias Sohn 2161c1e5e4 Merge branch 'stable-5.6'
* stable-5.6:
  Cygwin expects forward slashes for commands to be run via sh.exe
  Make Logger instances final
  Move array designators from the variable to the type

Change-Id: I9a5dc570deb478525bf48ef526d8cba5b19418bf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-28 23:53:32 +01:00
Thomas Wolf 6f268f8ceb Cygwin expects forward slashes for commands to be run via sh.exe
FS_Win32_Cygwin replaces backslashes by / as a side-effect of
relativize(). When support for core.hooksPath was added, paths were
relativized in a different place using Path.resolve(), which doesn't
do that transformation. As a result hooks could not be run on Cygwin
in some cases.

Do the transformation in FS_Win32_Cygwin.runInShell(). In all other
places, File or Path objects are used, which give no guarantee about
the file separator (typically the system-dependent default separator),
so doing the transformation earlier still wouldn't guarantee that
sh.exe indeed gets a command string using forward slashes.

Bug: 558577
Change-Id: I3c07eb85f0ac7c5628a2e92f990e5cdb7ecf532f
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-02-28 17:18:23 -05:00
Lars Vogel 512ffc9e7f Using for-each loop in jdt
Running the JDT cleanup action for using a for-each loop on jgit

Change-Id: Ie724d8bbdff786ab0167089e90a9914a8135103c
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
2020-02-28 02:39:12 -05:00
David Pursehouse 14a157dff6 Make Logger instances final
Change-Id: Ibb997952917e47bc31a8cbe3863623bc959a8100
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-27 20:32:29 +09:00
David Pursehouse 5a6b6eee35 Move array designators from the variable to the type
As reported by Sonar Lint:

Array designators should always be located on the type for better code
readability. Otherwise, developers must look both at the type and the
variable name to know whether or not a variable is an array.

Change-Id: If6b41fed3483d0992d402d8680552ab4bef89ffb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-27 20:27:31 +09:00
Jonathan Nieder 6586c171b3 Merge "Revert "RevWalk: stop mixing lines of history in topo sort"" 2020-02-26 17:48:18 -05:00
Yunjie Li 7ba261fb5f ObjectWalk: Add null check before skip tree.
currVisit could be null if a blob is marked as start point in
ObjectWalk. Add null check before skipping current tree.

Change-Id: Ic5d876fe2800f3373d136979be6c27d1bbd38dc1
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-02-26 12:03:58 -08:00
Alex Spradlin e40c38ab08 Revert "RevWalk: stop mixing lines of history in topo sort"
This reverts commit b5e764abd2.

PlotWalk uses the TopoSortGenerator, which is causing problems for EGit users
who rely on the emission of commits being somewhat based on date as in the
previous topo-sort algorithm.

Bug: 560529
Change-Id: I3dbd3598a7aeb960de3fc39352699b4f11a8c226
Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
2020-02-26 14:47:20 -05:00
Lajos Olah 61ec1455c0 Do not fail if known hosts file does not contain valid host key
KnownHosts (implementing HostKeyRepository) in Jsch can return null
which could cause NullPointerException in Stream.of(...)

Change-Id: Iddcf5f34f8c8475a85ca7ae018bbe48d1b3fbbc0
Signed-off-by: Lajos Olah <lajos.olah.jr@gmail.com>
2020-02-26 03:57:59 -05:00
Matthias Sohn 25a6bd4d61 Merge branch 'stable-5.6'
* stable-5.6:
  Revert "Prepend hostname to subsection used to store file timestamp resolution"
  SimilarityRenameDetector: Fix inconsistent indentation
  Use indexOf(char) and lastIndexOf(char) rather than String versions
  Reorder modifiers to follow Java Language Specification
  GitmoduleEntry: Remove redundant import of class from same package
  Remove redundant "static" qualifier from enum declarations

Change-Id: Ibb66bef7e8373f81e3e653c9843d986243446d68
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-22 23:36:08 +01:00
Matthias Sohn abb461533f Revert "Prepend hostname to subsection used to store file timestamp resolution"
This reverts commit e102bbed99.

Resolving the hostname comes with a performance penalty. We no longer
store the timestamp resolution in the global git config which might be
copied around to other machines but in a dedicated jgit config meant for
automatically determined options like timestamp resolution. Hence there
is no strong reason anymore to have a hardware specific identifier in
the subsection name of file timestamp resolution options.

Bug: 560414
Change-Id: If8dcabe981eb1792db84643850faa6033f14b1cf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-22 22:57:12 +01:00
Michael Dardis 67b9effc65 Remove use of org.bouncycastle.util.encoders.Hex
Change-Id: I5c1ed0397ef99eb5d4f120da331b66c2d0f1707a
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-22 00:12:20 +01:00
Michael Dardis 4055522359 Remove use of org.bouncycastle.util.io.TeeOutputStream
Bug: 559106
Change-Id: Ife06f21a00d258780a2030745738194098c631cf
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-21 23:45:43 +01:00
Thomas Wolf f2b67be62a Make the IMatcher public API
Provide a static factory method to create a PathMatcher.

Bug: 559526
Change-Id: Ib7a4a1bcc658ac2f2a09d365b5b891669dfd7570
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-02-20 10:40:33 +01:00
David Pursehouse fdabbe67e2 SimilarityRenameDetector: Fix inconsistent indentation
Replace space indentation with tab indentation

Change-Id: Ic130d3bde5d3a73d8f5c6225974153573722d05b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 20:34:15 +09:00
David Pursehouse 6a72f2943d Use indexOf(char) and lastIndexOf(char) rather than String versions
An indexOf or lastIndexOf call with a single letter String can be
made more performant by switching to a call with a char argument.

Found with SonarLint.

As a side-effect of this change, we no longer need to suppress the
NON-NLS warnings.

Change-Id: Id44cb996bb74ed30edd560aa91fd8525aafdc8dd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:24:01 +09:00
David Pursehouse 064834d350 Reorder modifiers to follow Java Language Specification
The Java Language Specification recommends listing modifiers in
the following order:

1. Annotations
2. public
3. protected
4. private
5. abstract
6. static
7. final
8. transient
9. volatile
10. synchronized
11. native
12. strictfp

Not following this convention has no technical impact, but will reduce
the code's readability because most developers are used to the standard
order.

This was detected using SonarLint.

Change-Id: I9cddecb4f4234dae1021b677e915be23d349a380
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:15:43 +09:00
David Pursehouse 2d1acae6d8 GitmoduleEntry: Remove redundant import of class from same package
Change-Id: I613489a937dd53780b0abfd6b1bd0ed964a5ddd8
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:06:18 +09:00
David Pursehouse 9be93b7991 Remove redundant "static" qualifier from enum declarations
Nested enum types are implicitly static.

Change-Id: Id3d7886087494fb67bc0d080b4a3491fb4baac19
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-02-19 13:04:43 +09:00
Matthias Sohn 9e4781878b Merge branch 'stable-5.7'
* stable-5.7:
  Update API problem filter
  Prepare 5.6.2-SNAPSHOT builds
  JGit v5.6.1.202002131546-r
  Simplify ReftableCompactor

Change-Id: I1aa650e030fbaa7a1c898eef1836de529e0c30db
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-19 01:45:18 +01:00
Matthias Sohn 7a940542ee Merge branch 'stable-5.6' into stable-5.7
# By Matthias Sohn (3) and Han-Wen Nienhuys (1)
* stable-5.6:
  Update API problem filter
  Prepare 5.6.2-SNAPSHOT builds
  JGit v5.6.1.202002131546-r
  Simplify ReftableCompactor

Change-Id: I16ed174f9fc662934c3ebaea85a60690efbed1c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-19 01:44:18 +01:00
Alex Spradlin b5e764abd2 RevWalk: stop mixing lines of history in topo sort
The topological sort algorithm in TopoSortGenerator for RevWalk may mix
multiple lines of history, producing results that differ from C git's
git log whose man page states: "Show no parents before all of its
children are shown, and avoid showing commits on multiple lines of
history intermixed." Lines of history are mixed because
TopoSortGenerator merely delays a commit until all of its children have
been produced; it does not immediately produce a commit after its last
child has been produced.

Therefore, when the last child of a commit has been produced, unpop the
commit so that it will be returned upon the subsequent call to next() in
TopoSortGenerator. To avoid producing duplicates, mark commits that
have not yet been produced as TOPO_QUEUED so that when a commit is
popped, it is produced if and only if TOPO_QUEUED is set.

To support nesting with other generators that may produce the same
commit multiple times like DepthGenerator (for example, StartGenerator
does this), do not increment parent inDegree for the same child commit
more than once.

Modify tests that assert that TopoSortGenerator mixes lines of commit
history.

Change-Id: I4ee03c7a8e5265d61230b2a01ae3858745b2432b
Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
2020-02-18 15:17:23 -08:00
Matthias Sohn bc98a597ff Update API problem filter
Change-Id: Iced4e3cc68cc9c2a75f9bf5befd6db7ce2fb1374
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-14 09:17:07 +01:00
Matthias Sohn 2899483e96 Prepare 5.6.2-SNAPSHOT builds
Change-Id: Ie04d749bf16eea6db5e5c98be54ba093ca249d05
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-14 00:15:39 +01:00
Matthias Sohn 1b44f0e4a6 JGit v5.6.1.202002131546-r
Change-Id: I2807b9a9f7e8b877b72702ebd1f70c4f9f223481
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-13 21:45:37 +01:00
Han-Wen Nienhuys 753fbaf11a Simplify ReftableCompactor
The ReftableCompactor supported a byteLimit, but this is currently
unused. The FileReftableStack has a more sophisticated strategy that
amortizes compaction costs.

Rename min/maxUpdateIndex to reflogExpire{Min,Max}UpdateIndex to
reflect their purpose  more accurately.

Since reflogs are generally pruned chronologically (oldest entries are
expired first), one can only prune entries on full compaction, so they
should not be set by default.

Rephrase the function Reader#minUpdateIndex and maxUpdateIndex. These
vars are documented to affect log entries, but semantically, they are
about ref entries. Since ref entries have their timestamps
delta-compressed, it is important for the min/maxUpdateIndex values to
be coherent between different tables.

The logical timestamps for log entries do not have to be coherent in
different tables, as the timestamps of a log entry is part of the key.
For example, a table written at update index 20 may contain a tombstone
log entry at timestamp 1.

Therefore, we set ReftableWriter's min/maxUpdateIndex from the merged
tables we are compacting, rather than from the compaction settings
(which should only control reflog expiry.)

The previous behavior could drop log entries erroneously, especially
in the presence of tombstone log entries. Unfortunately, testing this
properly requires both an API for adding log tombstones, and a more
refined API for controlling automatic compaction. Hence, no test.

Change-Id: I2f4eb7866f607fddd0629809e8e61f0b9097717f
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2020-02-12 23:08:20 +01:00
Yunjie Li 34cf825ba6 Change the wildcard import to explicit ones.
Change-Id: Ia7bc1c4b6b82c127e1e4e0e657f69d249684e8e6
Signed-off-by: Yunjie Li <yunjieli@google.com>
2020-02-12 18:45:26 +09:00
Matthias Sohn 003e4cfc23 Prepare 5.3.8-SNAPSHOT builds
Change-Id: I64ef3b4cf30e87d058001c2c6f1458f4b0c1df00
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 11:54:11 +01:00
Matthias Sohn 0c3b425099 JGit v5.3.7.202002110540-r
Change-Id: I3ba41af516b9d3b74ac580b3e170ad0e6e0663d4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 11:37:34 +01:00
Matthias Sohn 425e2607fe Prepare 5.1.14-SNAPSHOT builds
Change-Id: Ic7d6f1a6c3123af1a1fa782f052b0cea3b6f28c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 11:12:20 +01:00
Matthias Sohn e0abe9e2a7 JGit v5.1.13.202002110435-r
Change-Id: Iedc8699ad1a24efe7ddb47ae919c75b9d36141d4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 10:30:36 +01:00
Matthias Sohn e68e0e7f03 Merge branch 'stable-5.6'
* stable-5.6:
  reftable: don't check deadline on the first try
  reftable: clarify comment
  reftable: clear cache on full compaction
  reftable: remove outdated comment
  reftable: clarify that LogCursor may return a null ReflogEntry

Change-Id: I9458a746311984fa687b3da964805e2568ed37f3
2020-02-11 01:39:27 +01:00
Han-Wen Nienhuys 79266a1fe5 reftable: don't check deadline on the first try
This helps debug stepping.

Change-Id: I020dafab4ffac75e6df0dbcde0ed4805c7867f72
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2020-02-11 00:03:33 +01:00
Han-Wen Nienhuys 936a031ca3 reftable: clarify comment
Change-Id: I16e32aeb325b604eb31f84db18a214f792395941
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 00:03:12 +01:00
Han-Wen Nienhuys 8c9f7656c3 reftable: clear cache on full compaction
The merged table contains handles to open files. A full compaction
causes those files to be closed, and so further lookups would fail
with EBADF.

Change-Id: I7bb74f7228ecc7fec9535b00e56a617a9c18e00e
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-11 00:03:09 +01:00
Han-Wen Nienhuys dd203f03c2 reftable: remove outdated comment
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I41a7d9934d53553b5f81f40ff9d663378676ac5d
2020-02-10 12:43:36 +01:00
Han-Wen Nienhuys 0bf2564433 reftable: clarify that LogCursor may return a null ReflogEntry
Change-Id: I1a4d5c262cd196dca37876aec00bb974a45e9fcd
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2020-02-10 12:43:36 +01:00
Matthias Sohn 009674e99f Merge branch 'stable-5.6'
* stable-5.6:
  Restore behavior of CloneCommand

Change-Id: I3092bf214c41436b57e0ede9d2202f8aabf15471
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-10 10:48:39 +01:00
Lars Vogel 9d1e477417 Use lambdas where possible
Using the batch cleanup operation via Source -> Cleanup -> "Use lambdas
where possible" from standard JDT.

Change-Id: I5452bd94fdccc920ade071228aeed3a8b9fdbe62
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
2020-02-05 06:11:06 -05:00
Thomas Wolf 75a80c5d3c Restore behavior of CloneCommand
Commit 6216b0de changed the behavior of the setMirror(),
setCloneAllBranches(), and setBranchesToClone() operations. Before
that commit, these could be set and reset independently and only in
call() it would be determined what exactly to do. Since that commit,
the last of these calls would determine the operation. This means
that the sequence

  cloneCommand.setCloneAllBranches(true);
  cloneCommand.setBranchesToClone(/* some list of refs */);

would formerly do a "clone all" giving a fetch refspec with wildcards

  +refs/heads/*:refs/remotes/origin/*

which picks up new upstream branches, whereas since commit 6216b0de
individual non-wildcard fetch refspecs would be generated and new
upstream branches would not be fetched anymore.

Undo this behavioral change. Make the operations independently settable
and resettable again, and determine the exact operation only in call():
mirror=true > cloneAll=true > specific refs, where ">" means "takes
precedence over", and if none is set assume cloneAll=true.

Note that mirror=true implies setBare(true).

Bug: 559796
Change-Id: I7162b60e99de5e3e512bf27ff4113f554c94f5a6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2020-02-04 08:49:07 +01:00
Joshua Redstone 2a134e1541 AmazonS3: Speed up fetch, try recent packs first
When fetching remote objects, WalkFetchConnection searches remote
packs in the order provided by WalkRemoteObjectDatabase:getPackNames.
Previously, for TransportAmazonS3, the packs were in no particular
order. This resulted in potential many extra calls to get pack idx
files.

This change modifies TransportAmazonS3 and AmazonS3 so that
getPackNames returns a list sorted with the most recently modified
packs first.  In the case of fetching recent changes to a repo,
this dramatically reduces the number of packs searched and speeds
up fetch.

Note: WalkRemoteObjectDatabase::getPackNames does not specify
the order of the returned names.

Testing: did "mvn clean install" in root dir and all tests passed.
And manually constructed some S3 repos and using jgit.sh
confirmed that the freshest pack was checked first.

Change-Id: I3b968fee825e793be55566e28c2d69d0cbe53807
Signed-off-by: Joshua Redstone <redstone@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-02 14:44:22 +01:00
Matthias Sohn eaff0a2eba Merge branch 'stable-5.7'
* stable-5.7:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I0538bcba259f7229790a602ac9de120464a1260d
2020-02-01 02:17:49 +01:00
Matthias Sohn 7e3e740cc7 Merge branch 'stable-5.6' into stable-5.7
* stable-5.6:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I79d615dff66493b60d3a4bcbdc57b9455e8d6673
2020-02-01 02:09:37 +01:00
Matthias Sohn 3d59d1b80c Merge branch 'stable-5.5' into stable-5.6
* stable-5.5:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I73d16b53df02bf735c2431588143efe225a4b5b4
2020-02-01 02:06:47 +01:00
Matthias Sohn 68b0645a2e Merge branch 'stable-5.4' into stable-5.5
* stable-5.4:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I76a62da98182f0c504b1ea8b7d37cecdf4eea7e0
2020-02-01 01:53:27 +01:00
Matthias Sohn 195b8fc94c Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I12002dbfed9dff14fc6d2df9787d92eab5b1fa78
2020-02-01 01:28:40 +01:00
Matthias Sohn 48dd3385ab Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows
  Replace usage of ArrayIndexOutOfBoundsException in treewalk
  Add config constants for WindowCache configuration options

Change-Id: I790098be00ff3f9b8278d54ae4fb7c11311816e9
2020-02-01 01:14:01 +01:00
David Pursehouse f57a7bd230 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix string format parameter for invalidRefAdvertisementLine
  WindowCache: add metric for cached bytes per repository
  pgm daemon: fallback to user and system config if no config specified
  WindowCache: add option to use strong refs to reference ByteWindows

Change-Id: I741059a1d0d5950ab5bc16ec70352655ee926a24
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2020-02-01 01:09:46 +01:00
David Pursehouse 7d5300023e Fix string format parameter for invalidRefAdvertisementLine
The externalized error message added in f4fc640 ("BasePackConnection:
Check for expected length of ref advertisement", Dec 18, 2019) uses a
malformed string format. Since there is only one formatting argument,
it should be referenced with '{0}' rather than '{1}'.

Change-Id: Ibda864dfb0bb902fe07ae4bba73117b212046e8a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2020-01-31 08:38:49 +09:00