Commit Graph

4844 Commits

Author SHA1 Message Date
Luca Milanesio 2dc572df24 Include size when comparing FileSnapshot
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).

Read and consider file size also, so that differing file size can help
to more accurately detect file changes without reading the file content.
Use bulk read to avoid multiple stat calls to retrieve file attributes.

Change-Id: I974288fff78ac78c52245d9218b5639603f67a46
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 14:39:14 +01:00
Luca Milanesio fef782128d Do not reuse packfiles when changed on filesystem
The pack reload mechanism from the filesystem works only by name
and does not check the actual last modified date of the packfile.

This lead to concurrency issues where multiple threads were loading
and removing from each other list of packfiles when one of those
was failing the checksum.

Rely on FileSnapshot rather than directly checking lastModified
timestamp so that more checks can be performed.

Bug: 544199
Change-Id: I173328f29d9914007fd5eae3b4c07296ab292390
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
2019-03-12 14:30:36 +01:00
Matthias Sohn d4078dccda Silence API warnings for new API introduced for fixes
Change-Id: I3ea7ff2efd33ca6c780afaef9010cec82780d7fa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 14:30:20 +01:00
Matthias Sohn 51f2979f13 Merge branch 'stable-5.3'
* stable-5.3:
  Reduce contention on PackFile.idx() function.
  Use SystemReader in JSchConfigSessionFactoryTest
  Avoid NPE in ObjectId.isId()

Change-Id: I1d13f6fb705258ae6d6e5fa5e733bfacd4f3d0e3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:14:14 +01:00
Matthias Sohn da8e47ddc1 Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  Reduce contention on PackFile.idx() function.

Change-Id: I6bf4c1db695b8fa134ea425bbd488d2dc5438152
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:13:06 +01:00
Matthias Sohn 17e9ec4544 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Reduce contention on PackFile.idx() function.

Change-Id: I3f981dd923209e4d2d23f3b10db9fda1e9e68104
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:12:04 +01:00
Matthias Sohn 57f3ee4c16 Merge branch 'stable-5.0' into stable-5.1
* stable-5.0:
  Reduce contention on PackFile.idx() function.

Change-Id: Ic50f375faa757076e2dfd6c25e9e0025482aa3d9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:10:59 +01:00
Matthias Sohn 35e96348d3 Merge branch 'stable-4.11' into stable-5.0
* stable-4.11:
  Reduce contention on PackFile.idx() function.

Change-Id: Ib1be8c04c9587c595f7d95df26f7be9b237bda40
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:09:45 +01:00
Matthias Sohn 495ecda567 Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
  Reduce contention on PackFile.idx() function.

Change-Id: I5dd7576018ab2e85d77d336f97c8e77ad71520c9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:08:34 +01:00
Matthias Sohn 47d7f7aabb Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
  Reduce contention on PackFile.idx() function.

Change-Id: I277e53aa752c8ffb8560de710d27ecb58871ec02
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-12 00:06:17 +01:00
Juergen Denner 19c4380689 Reduce contention on PackFile.idx() function.
In case of concurrent pack file access, threads may wait on the idx()
function even for already open files. This happens especially with a
slow file system.

Performance numbers are listed in the bug report.

Bug: 543739
Change-Id: Iff328d347fa65ae07ecce3267d44184161248978
Signed-off-by: Juergen Denner <j.denner@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-11 19:52:26 +01:00
Michael Keppler 391c7a25fa Avoid NPE in ObjectId.isId()
That method can easily be invoked with a null argument (e.g.
isId(repo.getFullBranch()), therefore it should handle null arguments.

Change was suggested in https://git.eclipse.org/r/#/c/137918/, which
tried to fix the same in egit only.

Bug:544982
Change-Id: I32d1df6e9b2946ab324eda7008721159019316b3
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
2019-03-09 19:54:29 -05:00
Michael Keppler 90e614a41d Fix Javadoc typo
Change-Id: I3f169eaf649843e954762bb4ac8f72fa5f88a67a
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
2019-03-08 14:43:30 +01:00
Jonathan Nieder ee5f449bca UploadPack: Do not retain commit body when checking wants
The commit body contains the commit message, which is not needed for
reachability checks.

Change-Id: Ie209c3b3f022579942f05b8b5d0625ce26400a5d
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-03-07 19:58:35 -08:00
Ivan Frade a33847b5d9 FetchV2Request.getWantedRefs: Bump @since to 5.4
The visibility change was merged after 5.3 release. The @since line
is wrong.

Fixing it in a new commit looks like the easiest way forward.

Change-Id: I6f72a9e684e99a4440cda1eb532c22b4b7bbdea5
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-03-07 17:38:39 -08:00
Jonathan Nieder 94c8fd55e3 Merge "FetchV2Request: make getWantedRefs public" 2019-03-07 17:28:05 -05:00
Matthias Sohn f8a632657e Merge "Prepare 5.4.0-SNAPSHOT builds" 2019-03-07 11:16:46 -05:00
Matthias Sohn 90107d53eb Prepare 5.4.0-SNAPSHOT builds
Change-Id: I90a4791f63d0eba23da744c720e869f1830b86e7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-07 00:34:14 +01:00
Minh Thai cc2a223fea Do not retain commit body in RevWalk for reachability checks
Commit body contains the message that is not needed for reachability checks, and
takes up memory unnecessarily.

Change-Id: I0c7f6da249bf9c4fda9dc9e62e809322c68effce
Signed-off-by: Minh Thai <mthai@google.com>
2019-03-06 15:29:59 -08:00
Matthias Sohn 0295ee1164 Prepare 5.3.0-SNAPSHOT builds
Change-Id: I9b934bcfad3d2091f4da163170348b0c8ef5e732
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-06 22:26:03 +01:00
Matthias Sohn aca889f8cc JGit v5.3.0.201903061415-rc1
Change-Id: I85866af97fe1c0c0a0da83a7605484082fa56b52
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-06 20:03:00 +01:00
Terry Parker e48410ae9a Track object inflation time in DfsReaderIoStats
This can help track down poor long tail performance that isn't accounted
for in the readIdxMicros or readBlockMicros metrics.

Change-Id: I701b9cfcc124f4ddb860d1766a11ea3557e604ce
Signed-off-by: Terry Parker <tparker@google.com>
2019-03-06 10:54:52 -08:00
David Pursehouse 01cf73c7ab Merge branch 'stable-4.11' into stable-5.0
* stable-4.11:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: If75b149e693005dd3fe06b523e6e6784bedf44c1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-06 14:15:36 +09:00
David Pursehouse 3fc9d43339 Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: I9052e318b5d920770f7c7121d36e3c58df9d5f5a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-06 14:14:40 +09:00
David Pursehouse ae5ea80363 Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: I0847251eb010616a705e0b91df4bdebc225fa95d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-06 14:13:48 +09:00
Matthias Sohn 7b3ee6f62e Fix error log message in ObjectDirectory.handlePackError()
Change-Id: I154f392ad025c4b642eb1123d375a0afaa853885
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit 997d785418)
2019-03-06 02:36:38 +01:00
Matthias Sohn e8f7bb3608 Properly format pack checksums in PackFile.idx()
Change-Id: Id805850dbe9a3d633168f3056e06ddeafd86f961
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit a33e4dc58a)
2019-03-06 02:36:38 +01:00
Matthias Sohn e8bd9556c0 Cancel gc if thread was interrupted
see
https://groups.google.com/d/msg/repo-discuss/oDB2rl3doDc/tFEh5Xt0CAAJ

Change-Id: Ia6d4631c64e065d8b9b09e0b45e7a9ea8ac3f41d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit 882fed0d96)
2019-03-06 02:36:38 +01:00
Luca Milanesio daefa69502 PackFile: report correct message for checksum mismatch
When the packfile checksum does not match the expected one
report the correct checksum error instead of reporting that
the number of objects is incorrect.

Change-Id: I040f36dacc4152ae05453e7acbf8dfccceb46e0d
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit 436c99ce59)
2019-03-06 02:36:38 +01:00
David Pursehouse e6fd4732d0 ObjectDirectory: Clean up logging
Externalize the message and log the pack file with absolute path.

Change-Id: I019052dfae8fd96ab67da08b3287d699287004cb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit 9665d86ba1)
2019-03-06 02:36:37 +01:00
Luca Milanesio baf1bb20d0 ObjectDirectory: extra logging on packfile exceptions
Display extra logging, including the exception with the associated
stacktrace, whenever a packFile can't be read and thus removed
from the packlist.

Change-Id: I97a4e31dc427bfcc0baae438dcbe2dcd4704b824
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
(cherry picked from commit 962babc4b2)
2019-03-06 02:36:37 +01:00
David Pursehouse 52179ee247 Merge branch 'stable-5.2'
* stable-5.2:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository

Change-Id: Ib972641105cef8089791fc65389c4f43e218620a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-05 10:24:34 +09:00
David Pursehouse c2dbdabfb9 Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  Fix error log message in ObjectDirectory.handlePackError()
  Properly format pack checksums in PackFile.idx()
  Cancel gc if thread was interrupted
  PackFile: report correct message for checksum mismatch
  ObjectDirectory: Clean up logging
  Bazel: Stop using native.git_repository

Change-Id: I42b72a3bba3b4c9389d91a1a35a8004836567e7c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-05 10:23:38 +09:00
Matthias Sohn 997d785418 Fix error log message in ObjectDirectory.handlePackError()
Change-Id: I154f392ad025c4b642eb1123d375a0afaa853885
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-04 13:51:12 +01:00
Matthias Sohn a33e4dc58a Properly format pack checksums in PackFile.idx()
Change-Id: Id805850dbe9a3d633168f3056e06ddeafd86f961
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-04 13:28:40 +01:00
Matthias Sohn 882fed0d96 Cancel gc if thread was interrupted
see
https://groups.google.com/d/msg/repo-discuss/oDB2rl3doDc/tFEh5Xt0CAAJ

Change-Id: Ia6d4631c64e065d8b9b09e0b45e7a9ea8ac3f41d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-03 18:35:04 +01:00
Luca Milanesio 436c99ce59 PackFile: report correct message for checksum mismatch
When the packfile checksum does not match the expected one
report the correct checksum error instead of reporting that
the number of objects is incorrect.

Change-Id: I040f36dacc4152ae05453e7acbf8dfccceb46e0d
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-03-03 17:52:33 +01:00
David Pursehouse 9665d86ba1 ObjectDirectory: Clean up logging
Externalize the message and log the pack file with absolute path.

Change-Id: I019052dfae8fd96ab67da08b3287d699287004cb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-03-03 13:15:19 +09:00
Dave Borowitz ea54b6a5a7 ResolveMerger: Destroy TemporaryBuffer on unchecked exceptions
Previously, we called destroy() to delete the temp file on failure only
when catching an IOException, not a RuntimeException. Use a slightly
different construction with a finally block to ensure it's always
deleted on error (assuming the JVM is still healthy enough).

Change-Id: Ie201f3cfc81099ee1cafed066632da76223cef1f
2019-03-01 12:35:38 -08:00
Ivan Frade e1b1e01bec FetchV2Request: make getWantedRefs public
There are valid cases where a hook, invoked by ProtocolV2Hook
and probably implemented in a different package, is interested
in knowing the wanted refs in the request.

Increase visibility to public in the wanted-ref method.

Change-Id: I5da085ac7af4c396c1cb85e630f40a57fc70b33e
Signed-off-by: Ivan Frade <ifrade@google.com>
2019-02-28 23:49:19 -08:00
Terry Parker 875dccf33c Expose the filter blob limit in UploadPack
Similar to UploadPack.getDepth() to know the shallow clone depth, expose
the user-specified filter blob limit for partial clones.

Change-Id: I04bde06862a1cf8a9862d950c15023c49d16a2a6
Signed-off-by: Terry Parker <tparker@google.com>
2019-02-28 16:00:24 -08:00
Thomas Wolf f491af08ea Merge changes from topic 'rebase_compatibility'
* changes:
  RebaseCommand: tighten check for --preserve-merges on --continue
  RebaseCommand: use orig-head to abort
  RebaseCommand: use orig-head in addition to head
2019-02-26 16:22:22 -05:00
Jonathan Nieder 2fe1f0d638 Merge "Strongly reference indices in DfsPackFile" 2019-02-26 15:08:39 -05:00
Gunnar Wagenknecht 98df7ec801 Merge "On Windows use %APPDATA%\gnupg as GPG directory if it exists" 2019-02-26 14:46:51 -05:00
Christian Halstrick 20b633d95d Merge "Remove duplicate externalized message" 2019-02-26 07:35:52 -05:00
Thomas Wolf ca011107b8 On Windows use %APPDATA%\gnupg as GPG directory if it exists
Hard-coding ~/.gnupg for the GPG directory doesn't work on Windows,
where GnuPG uses %APPDATA%\gnupg by default. Make the determination
of the directory platform-dependent.

Bug: 544797
Change-Id: Id4bfd39a981ef7c5b39fbde46fce9a7524418709
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 12:49:59 +01:00
Thomas Wolf 4169a95a65 Remove duplicate externalized message
Instead of a new "unexpectedNlinkValue" message use the already
existing "failedAtomicFileCreation". Remove a stray double quote
from the latter.

Change-Id: I1ba5e9ea48d3f7615354b2ace2575883070b3206
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 11:53:49 +01:00
Thomas Wolf 38e5a75ce4 RebaseCommand: tighten check for --preserve-merges on --continue
With native git, .git/rebase-merge/rewritten exists actually in two
different cases:

* as a file in git rebase --merge recording OIDs for copying notes
* as a directory in git rebase --preserve-merges

Add a comment, and check for isDirectory() instead of exists().

Bug: 511487
Change-Id: I6a3317b4234d4f41c41b3004cdc7ea0abf2c6223
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 16:18:27 +09:00
Thomas Wolf a64fa0bd7f RebaseCommand: fix ONTO_NAME, and --preserve-merges is interactive
ONTO_NAME must be "onto_name", not "onto-name".

For native git, --preserve-merges is an interactive mode. Create the
INTERACTIVE marker file, otherwise a native git rebase --continue
will fall back into rebase --merge mode before git 2.19.0 since it
started looking for the REWRITTEN directory to make the distinction
only then.[1]

This allows a JGit interactive rebase to be continued via native git
rebase --continue.

[1] https://github.com/git/git/commit/6d98d0c0

Bug: 511487
Change-Id: I13850e0fd96ac77d03fbb581c8790d76648dbbc6
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 16:18:27 +09:00
Thomas Wolf 4ad53c0cbf RebaseCommand: use orig-head to abort
Aborting a rebase used ORIG_HEAD to reset. Strictly speaking this is
not correct, since other commands run during the rebase (for instance,
when the rebase stopped on a conflict) might have changed ORIG_HEAD.

Prefer the OID recorded in the orig-head file, falling back to the
older "head" file if "orig-head" doesn't exist, and use ORIG_HEAD only
if neither exists.

Bug: 511487
Change-Id: Ifa99221bb33e4e4754377f9b8f46e76c8936e072
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 16:18:27 +09:00
Thomas Wolf 60cf85a4a3 Fix core.autocrlf for non-normalized index
With text=auto or core.autocrlf=true, git does not normalize upon
check-in if the file in the index contains already CR/LFs. The
documentation says: "When text is set to "auto", the path is
marked for automatic end-of-line conversion. If Git decides that
the content is text, its line endings are converted to LF on
checkin. When the file has been committed with CRLF, no conversion
is done."[1]

Implement the last bit as in canonical git: check the blob in the
index for CR/LFs. For very large files, we check only the first 8000
bytes, like RawText.isBinary() and AutoLFInputStream do.

In Auto(CR)LFInputStream, ensure that the buffer is filled as much as
possible for the isBinary() check.

Regarding these content checks, there are a number of inconsistencies:

* Canonical git considers files containing lone CRs as binary.
* RawText checks the first 8000 bytes.
* Auto(CR)LFInputStream checks the first 8096 (not 8192!) bytes.

None of these are changed with this commit. It appears that canonical
git will check the whole blob, not just the first 8k bytes. Also
note: the check for CR/LF text won't work with LFS (neither in JGit
nor in git) since the blob data is not run through the smudge filter.
C.f. [2].

Two tests in AddCommandTest actually tested that normalization was
done even if the file was already committed with CR/LF.These tests
had to be adapted. I find the git documentation unclear about the
case where core.autocrlf=input, but from [3] it looks as if this
non-normalization also applies in this case.

Add new tests in CommitCommandTest testing this for the case where
the index entry is for a merge conflict. In this case, canonical git
uses the "ours" version.[4] Do the same.

[1] https://git-scm.com/docs/gitattributes
[2] https://github.com/git/git/blob/3434569fc/convert.c#L225
[3] https://github.com/git/git/blob/3434569fc/convert.c#L529
[4] https://github.com/git/git/blob/f2b6aa98b/read-cache.c#L3281

Bug: 470643
Change-Id: Ie7310539fbe6c737d78b1dcc29e34735d4616b88
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 16:18:27 +09:00
Thomas Wolf b4c212309b RebaseCommand: use orig-head in addition to head
Since 2011-02-10 (i.e., git 1.7.6)[1] native git uses "orig-head" for
REBASE_HEAD. JGit was still using "head". Currently native git has a
legacy fall-back for reading this, but for how long? Let's write to
both. Note that JGit never reads this file.

[1] https://github.com/git/git/commit/84df4560

Bug: 511487
Change-Id: Id3742bf9bbc0001d850e801b26cc8880e646abfc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-26 16:18:27 +09:00
David Pursehouse e54243278f SHA1: Use externalized message in log
The non-externalized warning message says there is a "possible SHA-1
collision" but then the Sha1CollisionException is always thrown.

Replace the message with the existing externalised string that does
not say "possible".

Change-Id: I9773ec76b416c356e234a658fb119f98d33eac83
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-02-25 11:05:44 +09:00
David Pursehouse a5a62a4cdb JGitText: Remove unnecessary suffix from externalized message identifier
Change-Id: I7a3ade56c219826b81d33d1d52566a86ad2e853a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-02-25 11:05:44 +09:00
David Pursehouse 6305869bf1 FS_POSIX: Externalize log message
Change-Id: Ida538e05c4595fdce99f1fa87469a3be60160adb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-02-25 11:05:42 +09:00
Minh Thai 01f084f5d9 Strongly reference indices in DfsPackFile
DfsBlockCache.Ref might get cleared out if the JVM is running out of
memory. As a result, the index is not persisted for the request and
will be reloaded unnecessarily.

Change-Id: I3b57ad5e6673f77f2dc66177a649ac412a97fe20
Signed-off-by: Minh Thai <mthai@google.com>
2019-02-21 16:48:43 -08:00
David Pursehouse 2902255526 Merge branch 'stable-5.3'
* stable-5.3:
  ObjectDirectory: extra logging on packfile exceptions
  Prepare 5.3.0-SNAPSHOT builds
  JGit v5.3.0.201901162155-m1
  Fix tools/version.sh
  JGit v5.3.0.201901161700-m1

Change-Id: I3c511fbc08f21eec5bb49d443de33d43094da4a4
2019-02-18 11:22:31 +09:00
David Pursehouse 9d02e52406 Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: I4bbeb1d28ab042e3c6c74f7284f5d882ed034305
2019-02-18 11:21:38 +09:00
David Pursehouse 90876890bc Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
  ObjectDirectory: extra logging on packfile exceptions

Change-Id: Id2fea30664d53b31ad026ee5ae06a5a24d731a52
2019-02-18 11:20:56 +09:00
Matthias Sohn 3ae22bdf66 Add missing @since tag for new API method getUpdateIndex
Change-Id: Idd4a580e241a9105a55bc6f0457ad02a35e55faf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-02-17 20:58:25 -05:00
Matthias Sohn ae79969715 Fix NPE in PlotCommitList
Bug: 544513
Change-Id: I2fca766bbdc441a2dca947fba5dde532de7a10fd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-02-18 01:25:12 +01:00
David Pursehouse 0368906353 CommitBuilder: Deprecate setEncoding(String)
Deprecate the method in favor of setEncoding(Charset).

Update the only caller in the code base that was still using
the deprecated variant.

Change-Id: I6357f2d0c727007013c72e9d5b7c72a3f5f3f2b1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-02-18 00:14:35 +01:00
David Pursehouse e5c4057467 CommitBuilder: Add missing periods on methods' Javadoc
Change-Id: Ibcf64b23de8534f7831d6e40ee4b0ad2bade74ce
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-02-18 00:14:34 +01:00
Luca Milanesio 962babc4b2 ObjectDirectory: extra logging on packfile exceptions
Display extra logging, including the exception with the associated
stacktrace, whenever a packFile can't be read and thus removed
from the packlist.

Change-Id: I97a4e31dc427bfcc0baae438dcbe2dcd4704b824
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
2019-02-14 10:44:00 +09:00
Minh Thai 2b7fa04dad Fix bug in copyPackBypassCache's skip 'PACK' header logic
Bug caused the pack to be 12 bytes short when cold cache.  Also added
test for copyPackAsIs method.

Change-Id: Idf8fb0e50d1215245d4b032e2e00df4b218c115f
Signed-off-by: Minh Thai <mthai@google.com>
2019-02-13 12:48:05 -08:00
Thomas Wolf a29d11775c Atomic file creation: hard-linking may not be allowed
Android for instance forbids hard linking via a SELinux
policy. If we can't hard link, the NFS work-around for
atomic file creation cannot work at all. In this case,
fall back to not using the hard-linking mechanism.

Android throws an AccessDeniedException, so we catch that.
The javadoc on Files.createLink() indicates that another
possibility might be a SecurityException, so catch that,
too.

Bug: 543956
Change-Id: I551b7a45f7b2fbbd8cf94f0b7233dbd8a200520e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-02-02 23:23:46 +01:00
Matthias Sohn 8cad84ccfb Fix GC.deleteEmptyRefsFolders
This method tried to iterate spurious files which may exist in the
.git/refs folder, e.g. on Mac a .DS_Store may have been created there by
inspecting the folder using the finder application. This led to a
NotDirectoryException when deleteEmptyRefsFolders tried to create an
iterator for such a file entry. Skip files contained in the refs folder
to ensure the method only tries to iterate contained folders but not
files.

Change-Id: I5f31e733072a35db1e93908a9c69a8891ae5c206
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-28 02:04:45 -05:00
Thomas Wolf e7b4d108e1 Enable cloning only specific tags
Single-branch-clone should be able to clone a single tag. Enhance
CloneCommand to accept also full refs of tags in setBranchesToClone().
Make sure we also include fetch ref specs for the fetch command for
tags. This mimics the behavior of native git's single-branch clone:
git clone --branch <tag> --single-branch <URI>

Bug: 542611
Change-Id: I285cf043751d9b0ba71258ee8214c0e5d1191428
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-01-27 22:56:06 +01:00
David Pursehouse afb6339314 Bazel: Format BUILD file with buildifier
$ buildifier -lint=fix BUILD

Change-Id: Ie2e069db3936ce6a85c4c29e6a0137cda5e21f09
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-24 11:43:37 +09:00
Thomas Wolf 447e107069 RenameBranchCommand: more consistent handling of short ref names
Several problems:

* The command didn't specify whether it expected short or full names.
* For the new name, it expected a short name, but then got confused
  if tags or both local and remote branches with the same name existed.
* For the old name, it accepted either a short or a full name, but
  again got confused if a short name was given and a tag with the
  same name existed.

With such an interface, one cannot use Repository.findRef() to
reliably find the branch to rename. Use exactRef() for the new
name as by the time the Ref is needed its full name is known.
For determining the old Ref from the name, do the resolution
explicitly: first try exactRef (assuming the old name is a full
name); if that doesn't find anything, try "refs/heads/<old>" and
"refs/remotes/<old>" explicitly. Throw an exception if the name
is ambiguous, or if exactRef returned something that is not a
branch (refs/tags/... or also refs/notes/...).

Document in the javadoc what kind of names are valid, and add tests.

A user can still shoot himself in the foot if he chooses exceptionally
stupid branch names. For instance, it is still possible to rename a
branch to "refs/heads/foo" (full name "refs/heads/refs/heads/foo"),
but it cannot be renamed further using the new short name if a branch
with the full name "refs/heads/foo" exists. Similar edge cases exist
for other dumb branch names, like a branch with the short name
"refs/tags/foo". Renaming using the full name is always possible.

Bug: 542446
Change-Id: I34ac91c80c0a00c79a384d16ce1e727c550d54e9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-01-23 19:10:36 +01:00
Gunnar Wagenknecht c4420d2434 Allow to check for signing key
The new API is intended for UIs to check if signing will be possible or
would fail

Bug: 543579
Change-Id: I6ce1fd4210e46d49dcdf420c99d08c93e022136c
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
2019-01-22 03:38:36 -05:00
Thomas Wolf 55966dc592 Handle premature EOF in BundleFetchConnection
BundleFetchConnection.readLine() must abort on EOF, otherwise
it gets stuck in an endless loop.

Bug: 543390
Change-Id: I4cb3428560277888af114b928950d620bb6564f9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2019-01-22 01:49:03 -05:00
David Pursehouse a0f69ff387 IO: Open TemporaryBuffer.Heap in try-with-resource
Change-Id: I78a947fd1263b47b3df17bcc6e9b32497e68dd4a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-21 14:19:29 +09:00
David Pursehouse d11a40a14c Scanner: Open RevWalk in try-with-resource
Change-Id: Ieeec292c1346edbda2b26f021e75cbc1435bb9f9
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-21 13:45:58 +09:00
David Pursehouse b7810bf5a8 RawParseUtils: Avoid import of java.nio.charset.StandardCharsets
The import is only needed because of a reference to it in the Javadoc,
and can be avoided by explicitly specifying the package instead, which
is how it's referenced in other cases (Constants, FileHeader).

Change-Id: I0c6254a9adf1f52fb8f2c04a858b11696ad264f5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-20 20:27:12 +09:00
David Pursehouse 91b1fe4687 UploadPack: Suppress false-positive resource leak warning
Change-Id: Ida09cabb11740e6752a6f079d0528d38021fa2dd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2019-01-20 17:41:59 +09:00
Matthias Sohn 23affa6c77 Prepare 5.3.0-SNAPSHOT builds
Change-Id: I9f0bf3ff2b26f4a8300c98206beb19c2b9d4946a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-17 04:23:42 +01:00
Matthias Sohn 05159ed656 JGit v5.3.0.201901162155-m1
Change-Id: I350925ff3b1fba2c83a448bc4e6d67368573693e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-17 04:17:18 +01:00
Matthias Sohn 8fc2aef89e JGit v5.3.0.201901161700-m1
Change-Id: I935cae5ae09ee5598ebdf6d9209048afec0d42dc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-17 02:39:39 +01:00
Medha Bhargav Prabhala 137e91a465 Implement signing commits using BouncyCastle
This also includes a change to generating the jgit CLI jar. Shading is
no longer possible because it breaks the signature of BouncyCastle.
Instead, the Spring Boot Loader Maven plug-in is now used to generate an
executable jar.

Bug: 382212
Change-Id: I35ee3d4b06d9d479475ab2e51b29bed49661bbdc
Also-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Medha Bhargav Prabhala <mprabhala@salesforce.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-16 10:05:10 +01:00
Gunnar Wagenknecht 2343c688b1 Allow CommitCommand to sign commits
This change introduces the concept of a GpgSigner which will sign
commits. The GpgSigner will be of a specific implementation (eg.,
Bouncycastle or OpenPgP executable). The actual implementation is not
part of this change.

Bug: 382212
Change-Id: Iea5da1e885c039e06bc8d679d46b124cbe504c8e
Also-by: Medha Bhargav Prabhala <mprabhala@salesforce.com>
Signed-off-by: Medha Bhargav Prabhala <mprabhala@salesforce.com>
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
2019-01-16 10:05:09 +01:00
Jonathan Nieder 1131f35cd1 dfs: Remove synchronization in BlockBasedFile#LazyChannel
As explained in 'The "Double-Checked Locking is Broken"
Declaration'[*], Java's memory model does not support double-checked
locking:

 class LazyReadableChannel {
   private ReachableChannel rc = null;

   public ReadableChannel get() {
     if (rc == null) {
       synchronized (this) {
         if (rc == null) {
           rc = new ReadableChannel();
	 }
       }
     }
     return rc;
   }
 }

With JDK 5 and newer, there is a formal memory model that ensures this
works if "rc" is volatile, but it is still not thread-safe without
that.

Fortunately, this ReadableChannelSupplier is never passed between
threads, so it does not need to be thread-safe.  Simplify by removing
the synchronization.

[*] https://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

Change-Id: I0698ee6618d734fc129dd4f63fc047c1c17c94a9
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-01-14 10:55:28 -08:00
Jonathan Nieder 5f355ed4ad UploadPack: Read wanted refs in one shot
This allows scanning through refs once instead of once per ref, which
should make the lookup less expensive for some RefDatabase
implementations.

Change-Id: I1434f834186cc9a6b4e52659e692b1000c926995
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-01-11 11:58:08 -08:00
Jonathan Nieder ec94268fd4 Merge branch 'stable-5.2'
* stable-5.2:
  Prepare 5.2.2-SNAPSHOT builds
  JGit v5.2.1.201812262042-r
  Prepare 5.1.6-SNAPSHOT builds
  JGit v5.1.5.201812261915-r
  UploadPack: Filter refs used for deepen-not resolution
  UploadPack: Avoid calling AdvertiseRefsHook twice
  Prepare 5.1.5-SNAPSHOT builds
  JGit v5.1.4.201812251853-r
  UploadPack: Filter refs used for want-ref resolution
  UploadPack: Defer want-ref resolution to after parsing
  Call AdvertiseRefsHook for protocol v2
  Prepare 4.11.7-SNAPSHOT builds
  JGit v4.11.6.201812241910-r
  Prepare 4.9.9-SNAPSHOT builds
  JGit v4.9.8.201812241815-r
  UploadPack: Test filtering by AdvertiseRefsHook in stateless transports
  Prepare 4.7.8-SNAPSHOT builds
  JGit v4.7.7.201812240805-r
  Fix feature versions imported by feature org.eclipse.jgit.pgm
  Prepare 4.5.6-SNAPSHOT builds
  JGit v4.5.5.201812240535-r
  Call AdvertiseRefsHook before validating wants

Change-Id: Ia56348e54d62630d7c50a4747df89516fc5afad9
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-01-10 20:23:33 -08:00
Minh Thai 513d80db10 Lazily open ReadableChannel in BlockBasedFile.getOrLoadBlock
To avoid opening the readable channel in case of DfsBlockCache
hits. Also cleaning up typos around DfsBlockCache.

Change-Id: I615e349cb4838387c1e6743cdc384d1b81b54369
Signed-off-by: Minh Thai <mthai@google.com>
2019-01-10 14:22:54 -08:00
Matthias Sohn 23c30c6310 Fix warnings for usage of deprecated RefDatabase#getRef
Change-Id: I22a244114ef973b3801cf3b639ff782f21c02f98
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-08 18:41:11 -05:00
Minh Thai 4207b8d6ae DfsBlockCache: Consolidate where ReadableChannel is opened
Opening a readable channel can be expensive and the number of channels
can be limited in DFS. Ensure that caller of
BlockBasedFile.readOneBlock() is responsible for opening and closing
the file, and that the ReadableChannel is reused in the request. As a side
effect, this makes the code easier to read, with better use of
try-with-resources.

The downside is that this means a readable channel is always opened, even
when the entire pack is already available for copying from cache. This
should be an acceptable cost: it's rare enough not to overload the server
and from a client latency perspective, the latency cost is in the noise
relative to the data transfer cost involved in a clone. If this turns out
to be a problem in practice, we can reintroduce that optimization in a
followup change.

Change-Id: I340428ee4bacd2dce019d5616ef12339a0c85f0b
Signed-off-by: Minh Thai <mthai@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-01-08 14:31:56 -05:00
Minh Thai 8bc9acf264 DfsBlockCache to lock while loading object references
We see the same index being loaded by multiple threads. Each is
hundreds of MB and takes several seconds to load, causing server to
run out of memory.  This change introduces a lock to avoid these
duplicate works.  It uses a new set of locks similar in implementation
to the loadLocks for getOrLoad of blocks.  The locks are kept separate
to prevent long-running index loading from blocking out fast block
loading. The cache instance can be configured with a consumer to
monitor the wait time of the new locks.

Change-Id: I44962fe84093456962d5981545e3f7851ecb6e43
Signed-off-by: Minh Thai <mthai@google.com>
2019-01-08 09:59:40 -08:00
Christian Halstrick e406d500de Fix "jgit checkout -f" to overwrite dirty worktree files
CheckoutCommand had a setForce() method. But this didn't correspond
to native git's 'git checkout -f' option. Deprecate the old setForce()
method and move its implementation to a new method setForceRefUpdate()
and use it to implement the -B option in the CLI class Checkout.

Add a setForced() method and use it to fix the associated '-f' option of
the CLI Checkout class to behave like native git's 'git checkout -f'
which overwrites dirty worktree files during checkout.

This is still not fully matching native git's behavior: updating
additionally dirty index entries is not done yet.

Bug: 530771
Change-Id: I776b78eb623b6ea0aca42f681788f2e4b1667f15
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-08 10:01:09 +01:00
Mario Molina 6c8240a751 Return 'this' from setters in commands
To avoid breaking ABI, take the opportunity to give these setters
(hopefully sometimes better) names and deprecate their old names.

Change-Id: Ib45011678c3d941f8ecc1a1e0fdf4c09cdc337e3
Signed-off-by: Mario Molina <mmolimar@gmail.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-01-07 20:00:11 -05:00
Jonathan Nieder d83e85736e Don't swallow IOException
Swallowing intermittent errors and trying to recover from them
makes JGit's behavior hard to predict and difficult to debug.

Propagate the errors instead.  This doesn't violate JGit's usual
backward compatibility promise for clients because in these
contexts an IOException indicates either repository corruption or
a true I/O error.  Let's consider these cases one at a time.

In the case of repository corruption, falling back e.g. to an empty
set of refs or a missing ref will not serve a caller well.  The
fallback does not indicate the nature of the corruption, so they are
not in a good place to recover from the error.  This is analogous to
Git, which attempts to provide sufficient support to recover from
corruption (by ensuring commands like "git branch -D" cope with
corruption) but little else.

In the case of an I/O error, the best we can do is to propagate the
error so that the user sees a dialog and has an opportunity to try
again.  As in the corruption case, the fallback behavior does not
provide enough information for a caller to rely on the current error
handling, and callers such as EGit already need to be able to handle
runtime exceptions.

To be conservative, keep the existing behavior for the deprecated
Repository#peel method.  In this example, the fallback behavior is to
return an unpeeled ref, which is distinguishable from the ref not
existing and should thus at least be possible to debug.

Change-Id: I0eb58eb8c77519df7f50d21d1742016b978e67a3
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-01-07 14:00:28 -05:00
Jonathan Nieder 3f9a5ce13f Deprecate Repository#hasObject
Its implementation contains

		} catch (IOException e) {
			// Legacy API, assume error means "no"
			return false;
		}

Better to use ObjectDatabase#has, which throws IOException to report
errors.

Change-Id: I7de02f7ceb8f57b2a8ebdb16d2aa4376775ff933
Signed-off-by: Jonathan Nieder <jrn@google.com>
2019-01-04 17:21:20 -08:00
Michael Keppler d8bb1d09d2 Deprecate RawParseUtils.UTF8-CHARSET
That constant is just a redirection to a java standard constant
meanwhile. It is not referenced anymore in jgit code (and egit is just
removing it). Clients can use the redirection target directly.

Change-Id: I058d013f61da8d7b771c499d8743aafb8faa5ea8
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2019-01-04 03:05:39 +01:00
Christian Halstrick 530812d936 Fix detection of "initial checkout"
A checkout done directly after cloning (the "initial
checkout") has a different semantic as a default
checkout. That is defined in the documentation of
"git read-tree" [1]. JGit was detecting that it is
doing an initial checkout differently from native
git: jgit used to check that the index is empty
but native git required that the index file does
not exist [2]. Teach JGit to behave like native
git.

[1] https://github.com/git/git/blob/master/Documentation/git-read-tree.txt#L187
[2] https://marc.info/?t=154150811200001&r=1&w=2

Change-Id: I1dd0f1ede7cd7ea60d28607916d0165269a9f628
2019-01-02 21:18:29 -05:00
Michael Keppler 0d9e486714 Replace deprecated FirstLine by FirstCommand
and allow package org.eclipse.jgit.http.server to use package
org.eclipse.jgit.internal.transport.parser.

Change-Id: Ief330c3e75a735853d0a5a265a9ff56fb5128b99
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-12-31 16:31:37 +01:00
Jonathan Nieder dbf6f9f692 Move BaseReceivePack#advertisedRefs getter and setter to ReceivePack
Another step toward merging BaseReceivePack into ReceivePack.

Change-Id: If861e28ce512f556e574352fa7d4a0df0984693f
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-12-29 11:59:27 +01:00
Jonathan Nieder 939723807e Move BaseReceivePack#walk getter to ReceivePack
Another step toward merging BaseReceivePack into ReceivePack.

Change-Id: I43cf2e36e2d5b0cd85bf23c81469909c14757b63
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-12-29 11:59:27 +01:00
Jonathan Nieder aaf41a51b7 Move BaseReceivePack#db getter to ReceivePack
Another step toward eliminating BaseReceivePack as a separate API.

Change-Id: If7b7d5c65a043607a2424211adb479fa33a9077b
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-12-29 11:59:27 +01:00
Jonathan Nieder 937beaa1ae Move BaseReceivePack#pushCert getter and setter to ReceivePack
This is a first step toward eliminating the BaseReceivePack API.

Inspired by a larger change by Dan Wang <dwwang@google.com>.

Change-Id: I5c876a67d8db24bf808823d9ea44d991b1ce5277
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-12-29 11:59:26 +01:00
Matthias Sohn e78776fa5b Remove unused API problem filters
Change-Id: I8ed98ac60362ba071506b0bdacd2941ab72b6bf4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-12-29 11:55:02 +01:00