Commit Graph

2034 Commits

Author SHA1 Message Date
David Pursehouse 8212924ede PullCommandWithRebaseTest: Open File{Input|Output}Stream in try-with-resource
Change-Id: I1ff707ab7bab676603907f4c0bb1bc495503055b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-02-26 17:45:11 +09:00
David Pursehouse ca7d3e2734 RecursiveMergerTest: Open FileOutputStream in try-with-resource
Change-Id: I158333d6393fb807bc21fba23fec7ad474384471
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-02-26 17:39:57 +09:00
David Pursehouse 4651d01e29 PullCommandTest: Use try-with-resources for File{Input,Output}Stream
Change-Id: I09242eb289655c7554aefa9e0817d9b881db656b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-02-26 08:58:48 +09:00
David Pursehouse f0b46e1db4 Add ObjectIdSerializer
This is based on the ObjectIdSerialization class written by Shawn Pearce
for the Gerrit Code Review project in 2009 [1]. As mentioned in the
commit message there, it should be part of core JGit.

This implementation is slightly different to Shawn's version. Rather
than having separate methods for null/non-null ids, single methods are
implemented with @Nullable annotations.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/9792

Change-Id: I7599cf8bd1ecd546e2252783d6d672eb76804060
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-25 23:15:01 +01:00
David Pursehouse eba04da7be CheckoutCommandTest: Fix name of created temp file
Change-Id: I29048f83aee3848679bbc6ded09dd3dd4a2ea35b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-02-25 17:47:43 +09:00
David Pursehouse 5ef028ff4a IgnoreNodeTest: Suppress deprecation warnings in testEmptyIgnoreNode
Change-Id: I930c9408cc702af911419f2a7b3d03f652a2d4e3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-02-25 17:36:38 +09:00
Marc Strapetz 78420b7d0a Fix processing of gitignore negations
Processing of negated rules, like !bin/ was not working correctly: they
were interpreted too broad, resulting in unexpected untracked files
which should actually be ignored

Bug: 409664
Change-Id: I0a422fd6607941461bf2175c9105a0311612efa0
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2018-02-23 13:36:48 +01:00
Marc Strapetz 49cb6ba5dd PathMatcher: fix handling of **/
**/ should match only directories, but not files

Change-Id: I885c83e5912cac5bff338ba657faf6bb9ec94064
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2018-02-22 14:39:23 +09:00
David Pursehouse 0e20df710a Merge changes Id616611e,Ib6509e81,I52f5d3f2
* changes:
  RepoCommand: generate relative submodule URLs from absolute URLs.
  RepoCommand: don't record new commit if tree did not change
  RepoCommand: persist unreadable submodules in .gitmodules
2018-02-22 00:37:48 -05:00
Marc Strapetz 372e04dcf3 CGitIgnoreTest: also test untracked files
Change-Id: I21a4ebd63eaaa85aa2e68f99ef58c141189bdab4
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2018-02-21 14:50:50 -05:00
Han-Wen Nienhuys 6a420613f7 RepoCommand: generate relative submodule URLs from absolute URLs.
If a manifest file specifies an absolute URL on the same host on which
the superproject resides, rewrite the URLs to be relative.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Id616611e5195998fb665c8e7806539a3a02e219a
2018-02-21 13:41:50 +01:00
Han-Wen Nienhuys 3214171dec RepoCommand: don't record new commit if tree did not change
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Ib6509e816684256d723558d5e8f3c37de58a2ff8
2018-02-19 20:24:07 +09:00
Han-Wen Nienhuys 446a7096ef RepoCommand: persist unreadable submodules in .gitmodules
In cases where a manifest file mixes different remotes, a Gerrit
server process may not have access to all remotes, and won't be able
to produce a full submodule tree.

Preserving this information in .gitmodules will let downstream clients
reconstruct the full tree.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I52f5d3f288e771dca0af2b4dd3f3fa0f940dcf15
2018-02-19 20:24:07 +09:00
Matthias Sohn 282dc592da Merge "Merge branch 'stable-4.10'" 2018-02-14 18:18:43 -05:00
Matthias Sohn d2d22e1471 Merge branch 'stable-4.10'
* stable-4.10:
  Fix ssh host name handling for Jsch
  Jsch overrides the port in the URI with the one in ~/.ssh/config

Change-Id: I860fc61ceb12ae792b1ee7421046ecd32373b9f8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-14 23:28:28 +01:00
Matthias Sohn 6e0f8bacd4 Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
  Fix ssh host name handling for Jsch
  Jsch overrides the port in the URI with the one in ~/.ssh/config

Change-Id: Iff9076f65e767bbe8df016337b631bdaeb40ad98
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-14 21:25:16 +01:00
Thomas Wolf 185e53bce4 Fix ssh host name handling for Jsch
If we give Jsch access to the ssh config file, we must _not_ resolve
the host name from the alias. Instead we must give the alias (i.e.,
the host name as is in the URI) to Jsch, so that it finds the same
ssh config entry.

Otherwise if the hostname in the URI, which is taken as an alias in
ssh config ("Host" line), is unequal to the "Hostname" line, and
there happens to be another ssh config entry with that translated
host name as alias, Jsch will pick up that second entry, and we end
up with a strange mixture of both.

Add tests for this case.

Bug: 531118
Change-Id: I249d8c073b0190ed110a69dca5b9be2a749822c3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-02-14 13:39:28 +01:00
Thomas Wolf 891103609d Jsch overrides the port in the URI with the one in ~/.ssh/config
Jsch unconditionally overwrites the port from the ssh config
file (if a port is specified there), even if the URI explicitly does
give a different port.

Fix this, and add tests.

Change-Id: I7b014543c7ece26270e366db39d7647f82d64f0d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-02-14 13:32:37 +01:00
David Pursehouse f527d4548f InitCommand: Don't leave Repository open after Git is closed
The InitCommand returns a Git that is instantiated with the newly
created Repository, but the Repository is not closed with the Git
resulting in resource leaks.

Create the Git with `closeRepo` set to true, such that the Repository
is also closed when the Git is closed.

Adjust the tests to use try-with-resource on the Git instance.

Change-Id: Ib26e7428c7d8840956d1edb09e53b93e23e6fe5a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-02-14 14:09:40 +09:00
Matthias Sohn 19355ce123 URIish: add a test for equals and hashcode methods
Change-Id: I5727ac1757dca0d63631401bcb074ee8f58ce082
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-13 19:00:40 -05:00
Terry Parker 9530c10192 Add a minimum negotiation feature for fetch
Android an Chrome have several repos with >300k refs. We sometimes see
negotiations of >100k rounds. This change provides a "minimal negotiation"
feature on the client side that limits how many "have" lines the client
sends. The client extracts the current SHA-1 values for the refs in its
wants set, and terminates negotiation early when all of those values have
been sent as haves. If a new branch is being fetched then that set will
be empty and the client will terminate after current default minimum
of two rounds.

This feature is gated behind a "fetch.useminimalnegotiation" configuration
flag, which defaults to false.

Change-Id: Ib12b095cac76a59da6e8f72773c4129e3b32ff2b
Signed-off-by: Terry Parker <tparker@google.com>
2018-02-12 14:03:11 -08:00
Minh Thai 5efc3dcc6c MergedReftable to skip shadowed refs in same reftable
This would allow compact and GC process to clean up duplicate ref names in the reftables.

Change-Id: I2b9df0bf72dba63cc3525e374982e60559a776c2
Signed-off-by: Minh Thai <mthai@google.com>
2018-02-09 09:49:24 -08:00
Christian Halstrick 4d5231f001 Fix CleanCommand not to throw FileNotFoundExceptions
When CleanCommand is collecting the files and folders to be deleted
it may happen that the list of directories contains obsolete entries.
E.g. a folder and its parent folder may be in the list. Only the
parent folder would be sufficient.

This was a reason for hitting FileNotFoundExceptions when finally
trying to delete the files and folders. Improve CleanCommand
to ignore files to be deleted which are already gone.

Bug: 514434
Change-Id: I10caa01bfb9cec5967dfdaea50c6e4a713eeeabd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-08 10:27:21 +01:00
Hector Caballero 93654f75a2 GC: Remove empty references folders
After packaging references, the folders containing these references are
not deleted. In a busy repository, this causes operations to slow down
as traversing the references tree becomes longer.

Delete empty reference folders after the loose references have been
packed.
To avoid deleting a folder that was just created by another concurrent
operation, only delete folders that were not modified in the last 30
seconds.

Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: Ie79447d6121271cf5e25171be377ea396c7028e0
2018-02-07 18:37:12 -05:00
David Turner be2a8ed63b Basic submodule merge handling
This doesn't handle the really hard thing, which is merging spurious
conflicts inside .gitmodules files.  That's OK: git.git doesn't
either.  Users can resolve the conflict themselves and then commit
the merge.

Previously, jgit would crash when attempting to merge conflicting
submodule changes.  Even if there was no conflict, after a merge which
adds submodules, the repository would have been missing empty
directories for newly-added submodules.

This patch fixes the crash, and adds the empty directories where
necessary. It ensures that the index is in a conflicted state when
submodule changes conflict.

Reported-by: Alexey Korobkov
Bug: 494551
Change-Id: I79db6798c2bdcc1159b5b2589b02da198dc906a1
Signed-off-by: David Turner <dturner@twosigma.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-04 15:33:24 +01:00
Thomas Wolf 9ac415aa95 Fix index blob for merges with CRLF translations
Commit fc7d407 corrected line endings for working tree files resulting
from merges when CRLF translations are to be done. However, that also
resulted in the file content being put as-is into the index, which is
wrong. The index must contain the file content with reverse CRLF
translations applied.

With core.autocrlf=true, the working tree file should have CR-LF, but
the index blob must still contain only LF.

Fix this oversight and apply the inverse translation when updating the
index, similar to what is done in AddCommand.

Bug: 499615
Change-Id: I3a33931318bdb580b2390f3450f91ea8f258a6a4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-02-02 08:16:13 +01:00
David Pursehouse 0e4b3023e3 Fix version of slf4j in manifest files
Consistently require 1.7.0. We ship 1.7.2 with our p2 repository but
there is no need to require 1.7.2 since it should be API compatible with
1.7.0.

Change-Id: I8467bb14316cb24daa79e89275332107d2716190
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-02 01:12:20 +01:00
Thomas Wolf fc7d407d0b Honor CRLF settings when writing merge results
Merges are performed using the raw text as stored in the git
repository. When we write the merge result, we must apply the
correct CRLF settings. Otherwise the line endings in the result
will be wrong.

Bug: 499615
Change-Id: I37a9b987e9404c97645d2720cd1c7c04c076a96b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-02-02 00:45:32 +01:00
Thomas Wolf e12c6b58ee Minor improvements in git config file inclusions
* Section and key names in git config files are case-insensitive.
* If an include directive is invalid, include the line in the
  exception message.
* If inclusion of the included file fails, put the file name into
  the exception message so that the user knows in which file the
  problem is.

Change-Id: If920943af7ff93f5321b3d315dfec5222091256c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-01-28 16:13:04 +01:00
Matthias Sohn 28aa3be21b Merge branch 'stable-4.10'
* stable-4.10:
  Revert handling of ssh IdentityFile to pre-4.9 behavior
  Prepare 4.10.1-SNAPSHOT builds
  JGit v4.10.0.201712302008-r

Change-Id: I13797f064de919c257feb8e96da226d7b1faaf7a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-01-13 01:06:42 +01:00
Matthias Sohn e72b2771a0 Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
  Revert handling of ssh IdentityFile to pre-4.9 behavior

Change-Id: I7cd2f0fe93be5193d6de2a376a023bdece5d84a9
2018-01-13 00:51:28 +01:00
Thomas Wolf bea3b46678 Revert handling of ssh IdentityFile to pre-4.9 behavior
Jsch caches keys (aka identities) specified in ~/.ssh/config via
IndentityFile only for the current Jsch Session. This results in
multiple password prompts for successive sessions.

Do the handling of IdentityFile exclusively in JGit, as it was before
4.9. JGit uses different Jsch instances per host and caches the
IdentityFile there, allowing it to be re-used in different sessions
for the same host.

* Add comments to explain this.
* Move the JschBugFixingConfig from OpenSshConfig to
  JschConfigSessionFactory to have all these Jsch work-arounds
  in one place.
* Make that config hide the IdentityFile config from Jsch to avoid
  that Jsch overrides the JGit behavior.

Bug: 529173
Change-Id: Ib36c34a2921ba736adeb64de71323c2b91151613
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-01-12 12:56:00 +01:00
David Pursehouse 1842c70b8d PackInserterTest#newLargeBlob: Factor Random instance out to class member
Instead of instantiating a new Random on each invocation of newLargeBlob,
create it once and reuse it.

This fixes a warning raised by Spotbugs about the Random object being
created and only used once.

Change-Id: I5b8e6ccbbc92641811537808aed9eae2034c1133
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-01-11 14:23:21 +09:00
Matthias Sohn 5a4b6fd237 Prepare 4.11.0-SNAPSHOT builds
Change-Id: I5d5e2befcf530d93457d44684bd9e4fc2392e5eb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-31 04:05:07 +01:00
Matthias Sohn 757fecd697 Prepare 4.10.1-SNAPSHOT builds
Change-Id: I139fb660c4630d9d073eabf37ff26ea3a823bcbd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-31 03:22:35 +01:00
Matthias Sohn 55bf5a6c11 JGit v4.10.0.201712302008-r
Change-Id: I9cc3a839d906acd01829df1de64cfafca8a6d008
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-31 02:08:55 +01:00
David Turner 243fba9a0a Add a command to deinitialize submodules
Change-Id: Iaaefc2cbafbf083d6ab158b1c378ec69cc76d282
Signed-off-by: David Turner <dturner@twosigma.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-27 23:47:25 +01:00
Marc Strapetz 8cc783ca7d URIish: support for empty ports
Properly parse URLs like "ssh://host:/path"

Bug: 519187
Change-Id: I0054868e30509e4ba919444be16c2a20f741545a
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2017-12-23 21:18:12 +01:00
Karsten Thoms 3832527b02 Add header Automatic-Module-Name for Java 9
Bug: 529075
Change-Id: I4532ce2c80eb91531d46026676502d636ccda706
Signed-off-by: Karsten Thoms <karsten.thoms@itemis.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-23 01:00:01 +01:00
Matthias Sohn 32775124d1 Merge branch 'stable-4.9'
* stable-4.9:
  Strings#convertGlob: fix escaping of patterns like [\[].

Change-Id: I18d55537002b3153db35f8a6b60f2f5317d17248
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-23 00:36:54 +01:00
Dmitry Pavlenko 279eaf0690 Strings#convertGlob: fix escaping of patterns like [\[].
Originally the patterns were escaped twice leading
to wrong matching results.

Bug: 528886
Change-Id: I26e201b4b0ef51cac08f940b76f381260fa925ca
Signed-off-by: Dmitry Pavlenko <pavlenko@tmatesoft.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-21 18:33:22 -05:00
Dave Borowitz e76a070e67 ConfigTest: Add some additional comment parsing tests
Change-Id: I0f1df93bd1ceeb847771ea6484f9de539e8c177b
2017-12-21 09:30:52 -05:00
Dave Borowitz be224f8d4c Config: Drop backslash in invalid escape sequences in subsections
These are ignored by C git when parsing:

$ git config -f - --list <<EOF
[foo "x\0y"]
bar = baz
[foo "x\qy"]
bar = baz
[foo "x\by"]
bar = baz
[foo "x\ny"]
bar = baz
[foo "x\ty"]
bar = baz
EOF
foo.x0y.bar=baz
foo.xqy.bar=baz
foo.xby.bar=baz
foo.xny.bar=baz
foo.xty.bar=baz

This behavior is different from value parsing, where an invalid escape
sequence is an error (which JGit already does as well):

$ git config -f - --list <<EOF
[foo]
bar = x\qy
EOF
fatal: bad config line 2 in standard input

Change-Id: Ifd40129b37d9a62df3d886d8d7e22f766f54e9d1
2017-12-21 09:30:52 -05:00
Dave Borowitz fa24425c60 Config: Match C git behavior more closely in escaping values
So this happened:

$ git config -f foo.config foo.a 'x"y'
$ git config -f foo.config foo.b x\\y
$ git config -f foo.config foo.c $'x\ny'
$ git config -f foo.config foo.d $'x\ty'
$ git config -f foo.config foo.e $'x\by'
$ cat foo.config
[foo]
        a = x\"y
        b = x\\y
        c = x\ny
        d = x\ty
        e = y

That last line is my shell interpreting the backspace byte:

$ python -c 'print repr(open("foo.config").read())'
'[foo]\n\ta = x\\"y\n\tb = x\\\\y\n\tc = x\\ny\n\td = x\\ty\n\te = x\x08y\n'

Change-Id: I1b059e1afb544c39ddb587c07bf79a06e99d0a09
2017-12-21 09:30:52 -05:00
Matthias Sohn 2464fa440f Fix remaining javadoc errors raised by doclint
For now ignore doclint "missing" warnings.

Change-Id: I0e5af7a757f4d92ffeeb113f30576a35414d6781
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-21 02:42:14 +01:00
Dave Borowitz c1a02f497a Merge branch 'stable-4.9'
* stable-4.9:
  PackInserter: Ensure objects are written at the end of the pack
  ObjectInserter: Add warning about mixing read-back with writes

Change-Id: I308e7c1c6b72e8d4d9b5d0f4f51e9815fc92d7d7
2017-12-20 14:39:11 -05:00
Dave Borowitz 43ef5dabf1 PackInserter: Ensure objects are written at the end of the pack
When interleaving reads and writes from an unflushed pack, we forgot to
reset the file pointer back to the end of the file before writing more
new objects. This had at least two unfortunate effects:
  * The pack data was potentially corrupt, since we could overwrite
    previous portions of the file willy-nilly.
  * The CountingOutputStream would report more bytes read than the size
    of the file, which stored the wrong PackedObjectInfo, which would
    cause EOFs during reading.

We already had a test in PackInserterTest which was supposed to catch
bugs like this, by interleaving reads and writes. Unfortunately, it
didn't catch the bug, since as an implementation detail we always read a
full buffer's worth of data from the file when inflating during
readback. If the size of the file was less than the offset of the object
we were reading back plus one buffer (8192 bytes), we would completely
accidentally end up back in the right place in the file.

So, add another test for this case where we read back a small object
positioned before a large object. Before the fix, this test exhibited
exactly the "Unexpected EOF" error reported at crbug.com/gerrit/7668.

Change-Id: I74f08f3d5d9046781d59e5bd7c84916ff8225c3b
2017-12-20 12:43:31 -05:00
Dave Borowitz 31a2d09c9c Config: Rewrite subsection and value escaping and parsing
Previously, Config was using the same method for both escaping and
parsing subsection names and config values. The goal was presumably code
savings, but unfortunately, these two pieces of the git config format
are simply different.

In git v2.15.1, Documentation/config.txt says the following about
subsection names:

  "Subsection names are case sensitive and can contain any characters
  except newline (doublequote `"` and backslash can be included by
  escaping them as `\"` and `\\`, respectively).  Section headers cannot
  span multiple lines.  Variables may belong directly to a section or to
  a given subsection."

And, later in the same documentation section, about values:

  "A line that defines a value can be continued to the next line by
  ending it with a `\`; the backquote and the end-of-line are stripped.
  Leading whitespaces after 'name =', the remainder of the line after
  the first comment character '#' or ';', and trailing whitespaces of
  the line are discarded unless they are enclosed in double quotes.
  Internal whitespaces within the value are retained verbatim.

  Inside double quotes, double quote `"` and backslash `\` characters
  must be escaped: use `\"` for `"` and `\\` for `\`.

  The following escape sequences (beside `\"` and `\\`) are recognized:
  `\n` for newline character (NL), `\t` for horizontal tabulation (HT,
  TAB) and `\b` for backspace (BS).  Other char escape sequences
  (including octal escape sequences) are invalid."

The main important differences are that subsection names have a limited
set of supported escape sequences, and do not support newlines at all,
either escaped or unescaped. Arguably, it would be easy to support
escaped newlines, but C git simply does not:

  $ git config -f foo.config $'foo.bar\nbaz.quux' value
  error: invalid key (newline): foo.bar
  baz.quux

I468106ac was an attempt to fix one bug in escapeValue, around leading
whitespace, without having to rewrite the whole escaping/parsing code.
Unfortunately, because escapeValue was used for escaping subsection
names as well, this made it possible to write invalid config files, any
time Config#toText is called with a subsection name with trailing
whitespace, like {foo }.

Rather than pile hacks on top of hacks, fix it for real by largely
rewriting the escaping and parsing code.

In addition to fixing escape sequences, fix (and write tests for) a few
more issues in the old implementation:

* Now that we can properly parse it, always emit newlines as "\n" from
  escapeValue, rather than the weird (but still supported) syntax with a
  non-quoted trailing literal "\n\" before the newline. In addition to
  producing more readable output and matching the behavior of C git,
  this makes the escaping code much simpler.
* Disallow '\0' entirely within both subsection names and values, since
  due to Unix command line argument conventions it is impossible to pass
  such values to "git config".
* Properly preserve intra-value whitespace when parsing, rather than
  collapsing it all to a single space.

Change-Id: I304f626b9d0ad1592c4e4e449a11b136c0f8b3e3
2017-12-18 17:46:37 -05:00
David Pursehouse 663cb669b1 PushConnectionTest: Increase maxCommandBytes yet again
It was already increased in 61a943e and 661232b but is still not
enough to take into account snapshot versions that are 100 or more
commits ahead of tag, i.e. 4.9.2.201712150930-r.105-gc1d37ca27

Change-Id: Ibeff73adae06b92fe5bb9c5eced9e4c6a08c437c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-18 10:22:33 +09:00
Matthias Sohn 1e56842742 Prepare 4.9.3-SNAPSHOT builds
Change-Id: Ife3f2b0b5407227f89ded42358adbf01d53e14cf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-16 03:49:03 +01:00
Matthias Sohn 24b7e91264 JGit v4.9.2.201712150930-r
Change-Id: I013964045d532659a4be3b81d6612b59bc9ffb14
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-15 15:29:36 +01:00
David Pursehouse c60814d1d5 ConfigTest: Remove redundant assignment
Change-Id: Ia913dbe6b7ad4b7000525fda8ca08cbc8dd87da6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-15 09:33:50 +09:00
David Pursehouse ec7f88eec8 Config: Remove the include functionality
The Config class must be safe to run against untrusted input files.
Reading arbitrary local system paths using include.path is risky for
servers, including Gerrit Code Review.

This was fixed on master [1] by making "readIncludedConfig" a noop
by default. This allows only FileBasedConfig, which originated from
local disk, to read local system paths.

However, the "readIncludedConfig" method was only introduced in [2]
which was needed by [3], both of which are only on the master branch.
On the stable branch only Config supports includes. Therefore this
commit simply disables the include functionality.

[1] https://git.eclipse.org/r/#/c/113371/
[2] https://git.eclipse.org/r/#/c/111847/
[3] https://git.eclipse.org/r/#/c/111848/

Bug: 528781
Change-Id: I9a3be3f1d07c4b6772bff535a2556e699a61381c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-15 08:58:00 +09:00
Shawn Pearce 3a7704638a Make Config.readIncludedConfig a noop by default
The Config class must be safe to run against untrusted input files.
Reading arbitrary local system paths using include.path is risky for
servers, including Gerrit Code Review.  Return null by default to
incide the include should be ignored.

Only FileBasedConfig which originated from local disk should be trying
to read local system paths.  FileBasedConfig already overrides this
method with its own implementation.

Change-Id: I2ff31753868aa1bbac4a6843a4c23e50bd6f46f3
2017-12-13 17:50:52 -08:00
Thomas Wolf cde32497b5 Merge "URIishTest: more Windows file-protocol tests" 2017-12-09 16:27:18 -05:00
Marc Strapetz 93462447b4 URIishTest: more Windows file-protocol tests
Change-Id: Id5fbd8bb9cd05da89d27e9532612d64ae84a55ba
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2017-12-09 12:15:12 +01:00
David Pursehouse c861c0e2ee PackInserterTest: Prevent potential NPE dereferencing Path.getFileName()
Path.getFileName() may return null if the path has zero elements.

Enclose the dereference in a null-check.

Change-Id: I7ea3d3f07edc13a80b593d28e8fd512a4e1ed56b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-08 17:21:52 +09:00
David Pursehouse fdacfaecc4 Specify consistent version range for junit in OSGi manifests
There are several different version ranges specified in the various
manifest files.

Align them all to the same range:  [4.12,5.0.0)

Change-Id: I02205b8b8546c9f53ed431b5fd9abf6ddcda4423
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-08 17:00:56 +09:00
David Pursehouse 171f84a041 Use constants from StandardCharsets instead of hard-coded strings
Instead of hard-coding the charset strings "US-ASCII", "UTF-8", and
"ISO-8859-1", use the corresponding constants from StandardCharsets.

UnsupportedEncodingException is not thrown when the StandardCharset
constants are used, so remove the now redundant handling.

Because the encoding names are no longer hard-coded strings, also
remove redundant $NON-NLS warning suppressions.

Also replace existing usages of the constants with static imports.

Change-Id: I0a4510d3d992db5e277f009a41434276f95bda4e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-07 20:02:59 +09:00
David Pursehouse df3a7c32a4 ConfigTest: Move pathToString to FileUtils
ConfigTest#pathToString is not visible to FileBasedConfigTest when
bulding with bazel.

Move it to FileUtils rather than messing about with the bazel build
rules to make it visible.

Change-Id: Idcfd4822699dac9dc4a426088a929a9cd31bf53f
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-06 09:34:07 +09:00
Marc Strapetz 26d78902f8 FileBasedConfig: support for relative includes
Relative include.path are now resolved against the config's parent
directory. include.path starting with ~/ are resolved against the
user's home directory

Change-Id: I91911ef404126618b1ddd3589294824a0ad919e6
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2017-12-04 23:38:24 +01:00
Marc Strapetz e1adfee5f5 ConfigTest: fix on Windows
Change-Id: I37a2ef611aef97faf1b891a9660c1745435a915d
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-04 23:38:22 +01:00
Matthias Sohn 68c77a4d39 Prepare 4.9.2-SNAPSHOT builds
Change-Id: I5879ad4aee94ff6783b5589728912117f2495dd3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-03 14:17:43 +01:00
Matthias Sohn a3588cbb2a JGit v4.9.1.201712030800-r
Change-Id: I8bf477778c9dac41cb65233a9e7d590531a836b7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-03 13:59:36 +01:00
David Pursehouse 8b159b6235 Make local assert methods private in test classes
Change-Id: I1bed28a1eac3c7f84cc40841853b9540c72be265
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-01 10:56:18 +09:00
Matthias Sohn 470629a237 Merge branch 'stable-4.9'
* stable-4.9:
  GC: Delete stale temporary packs and indexes

Change-Id: I49b37845ee8a465404b801a2d8de0205a2e7ba30
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-11-30 23:24:25 +09:00
Minh Thai 159da6dacc Break down DfsBlockCache stats by pack file extension.
Change-Id: Iaecf0580279b33e3e2439784528cae7b69fb28bc
Signed-off-by: Minh Thai <mthai@google.com>
2017-11-27 21:55:21 -08:00
Hector Caballero bac4d32d39 GC: Delete stale temporary packs and indexes
When a GC operation is interrupted, temporary packs and indexes can be
left on the pack folder. In big, busy repositories this can lead to
significant amounts of wasted disk space if this interruption is done
with a certain frequency.

Remove stale temporary packs and indexes at the end of the GC process so
they do not accumulate. To avoid interfering with a possible concurrent
JGit GC process in the same repository, only delete temporary files that
are older than one day.

Change-Id: If9b6c1e57fac8a6a0ecc0a703089634caba4caae
Signed-off-by: Hector Caballero <hector.caballero@ericsson.com>
2017-11-24 05:13:24 -05:00
Matthias Sohn f0c119de4f Merge branch 'stable-4.9'
* stable-4.9:
  Ignore warning for minor version change without API change
  Silence boxing warning
  Prepare 4.5.5-SNAPSHOT builds
  JGit v4.5.4.201711221230-r
  Fix LockFile semantics when running on NFS
  Honor trustFolderStats also when reading packed-refs
  Prepare 4.5.4-SNAPSHOT builds
  JGit v4.5.3.201708160445-r

Change-Id: Icc33d2e36f140e8714fce088379673a8834ae9de
2017-11-24 01:18:13 +01:00
Matthias Sohn 03abd1dff2 Ignore warning for minor version change without API change
- this is a new warning option in Eclipse 4.7 and higher
- we always change version of all bundles in a release to keep release
engineering simple

Change-Id: Ic7523d77b67b2802f1bab3bc70af250d712a034f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-24 01:12:14 +01:00
Matthias Sohn 7608de5e5d Prepare 4.5.5-SNAPSHOT builds
Change-Id: I71f946f2875716670a2d74c21a8ab38a1f53a25c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 19:27:22 +01:00
Matthias Sohn 6372c60ab2 JGit v4.5.4.201711221230-r
Change-Id: Ia1079da239c5b3fde1ba8d2acc4e465a46297b4d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 18:31:24 +01:00
David Pursehouse bd052b94aa Merge branch 'stable-4.9'
* stable-4.9:
  Yet another work-around for a Jsch bug: timeouts

Change-Id: I7cf227c62a3c06f91cee1a6c61719b6fe50da883
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-11-22 11:31:29 +09:00
Thomas Wolf 5284cc1bf7 Yet another work-around for a Jsch bug: timeouts
Jsch 0.1.54 passes on the values from ~/.ssh/config for
"ServerAliveInterval" and "ConnectTimeout" as read from
the config file to java.net.Socket.setSoTimeout(). That
method expects milliseconds, but the values in the config
file are seconds!

The missing conversion in Jsch means that the timeout is
set way too low, and if the server doesn't respond within
that very short time frame, Jsch kills the connection and
then throws an exception with a message such as "session is
down" or "timeout in waiting for rekeying process".

As a work-around, do the conversion to milliseconds in the
Jsch-facing Config interface of OpenSshConfig. That way Jsch
already gets these values as milliseconds.

Bug: 526867
Change-Id: Ibc9b93f7722fffe10f3e770dfe7fdabfb3b97e74
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-11-20 22:44:23 +01:00
Dave Borowitz 8b3ab4343c Config: Handle leading/trailing single whitespaces
Change-Id: I468106acd2006d0a174c76dfd4bce231f1c7a6f8
2017-11-20 13:55:25 -05:00
Shawn Pearce 7bf8f52699 Merge changes from topic 'includeDeletes'
* changes:
  Add flag for keeping ref tombstones in GC reftable
  Preserve ref tombstone when compact top retable stack
2017-11-16 10:45:57 -05:00
Minh Thai 15a189e4e0 Add flag for keeping ref tombstones in GC reftable
A tombstone will prevent a delayed reference update from resurrecting the
deleted reference.

Change-Id: Id9f4df43d435a299ff16cef614821439edef9b11
Signed-off-by: Minh Thai <mthai@google.com>
2017-11-15 22:48:04 -08:00
Matthias Sohn af2eaaed68 Remove unused import from ReftableCompactorTest
Change-Id: Ib6d7fb6b56a94be307b07fefacf5d9c77fce0447
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-15 16:22:55 +01:00
David Pursehouse ef12214a7c Merge "ObjectDirectory: Add pack directory getter" 2017-11-14 20:37:26 -05:00
Jonathan Nieder 295c5ea7d3 Merge "ReftableCompactor should accept 0 for minUpdateIndex" 2017-11-14 15:05:19 -05:00
Minh Thai 0e5abbfafc ReftableCompactor should accept 0 for minUpdateIndex
Do not use 0 as the unset value for minUpdateIndex, as input reftables
may have minUpdateIndex starting at 0.

Change-Id: Ie040a6b73d4a5eba5521e51d0ee4580713c84a3e
Signed-off-by: Minh Thai <mthai@google.com>
2017-11-14 10:50:24 -08:00
Hector Caballero 4334b27d3c ObjectDirectory: Add pack directory getter
So far, in order to get the pack directory it was necessary to resolve
it from the object directory. This resolution is already done when
creating the object directory, so simplify the call by just adding a
getter to the pack directory.

Change-Id: I69e783141dc6739024e8b3d5acc30843edd651a7
Signed-off-by: Hector Caballero <hector.caballero@ericsson.com>
2017-11-14 10:08:42 -05:00
Marc Strapetz 9bb126d12d FileUtils.toPath to convert File to Path
When invoking File.toPath(), an (unchecked) InvalidPathException may be
thrown which should be converted to a checked IOException.

For now, we will replace File.toPath() by FileUtils.toPath() only for
code which can already handle IOExceptions.

Change-Id: I0f0c5fd2a11739e7a02071adae9a5550985d4df6
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
2017-11-14 10:07:37 +01:00
Matthias Sohn c7fffc30d0 Remove an unused import from PackParserTest
Change-Id: I4182a1746b09dedab648e457d1ece6d667a01f12
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-11 01:01:03 +01:00
Jonathan Nieder b88204edfb Merge changes I22a8874b,I68ed4abd,I740bc4bf,Icbd17d15
* changes:
  BitmapWalker: do not revisit objects in bitmap
  Use bitmaps for non-commit reachability checks
  Make PackWriterBitmapWalker public
  UploadPackTest: construct commits in test method
2017-11-10 18:52:53 -05:00
Jonathan Tan d3021788d2 Use bitmaps for non-commit reachability checks
Currently, unless RequestPolicy#ANY is used, UploadPack rejects all
non-commit "want" lines unless they were advertized. This is fine,
except when "uploadpack.allowreachablesha1inwant" is true
(corresponding to RequestPolicy#REACHABLE_COMMIT), in which case one
would expect that "want"-ing anything reachable would work.

(There is no restriction that "want" lines must only contain commits -
it is allowed for refs to directly point to trees and blobs, and
requesting for them using "want" lines works.)

This commit has been written to avoid performance regressions as much
as possible. In the usual (and currently working) case where the only
unadvertized things requested are commits, we do a standard RevWalk in
order to avoid incurring the cost of loading bitmaps. However, if
unadvertized non-commits are requested, bitmaps are used instead, and
if there are no bitmaps, a WantNotValidException is thrown (as is
currently done).

Change-Id: I68ed4abd0e477ff415c696c7544ccaa234df7f99
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
2017-11-10 15:41:31 -08:00
Shawn Pearce 2ec71a7c0e Reject pack if delta exceeds array size limit
JGit's delta handling code requires the target to be a single byte
array. Any attempt to inflate a delta larger than fits in the 2GiB
limit will fail with some form of array index exceptions. Check for
this overflow early and abort pack parsing.

Change-Id: I5bb3a71f1e4f4e0e89b8a177c7019a74ee6194da
2017-11-09 09:27:54 -08:00
David Pursehouse 190b575be1 Suppress "Unlikely argument type for equals()" warnings in tests
This new warning was introduced in Eclipse 4.7 Oxygen [1].

The only instances of the warning are in test code that is asserting
that some class does not compare equal to Strings. As in the Gerrit
project [2] these asserts are arguably overkill, but arguably also
a reasonable test of an equals implementation. Ignore the warning in
these cases.

Note that if the project is opened in an earlier version of Eclipse,
a warning "Unsupported @SuppressWarnings" will be emitted.

[1] https://www.eclipse.org/eclipse/news/4.7/M6/
[2] https://gerrit-review.googlesource.com/#/c/gerrit/+/110339/

Change-Id: I08ea33d71e6009cf0f37e6492a475931f447256b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-11-06 10:34:34 +01:00
Jonathan Tan 5ea57ba1b5 UploadPackTest: construct commits in test method
In a subsequent commit, more tests will be added. This commit allows
those tests to reuse fields.

Change-Id: Icbd17d158cfe3ba4dacbd8a11a67f9e7607b41b3
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
2017-11-01 17:51:41 -07:00
David Pursehouse 651e17baca Merge branch 'stable-4.9'
* stable-4.9:
  PackInserter: Implement newReader()
  Move some strings from DfsText to JGitText
  FileRepository: Add pack-based inserter implementation
  ObjectDirectory: Factor a method to close open pack handles
  ObjectDirectory: Remove last modified check in insertPack

Change-Id: Ifc9ed6f5d8336bc978818a64eae122bceb933e5d
2017-11-02 08:30:01 +09:00
Dave Borowitz 678c99c057 PackInserter: Implement newReader()
Change-Id: Ib9e7f6439332eaed3d936f895a5271a7d514d3e9
2017-11-01 13:00:24 -04:00
Dave Borowitz f7ceeaa23f FileRepository: Add pack-based inserter implementation
Applications that use ObjectInserters to create lots of individual
objects may prefer to avoid cluttering up the object directory with
loose objects. Add a specialized inserter implementation that produces a
single pack file no matter how many objects. This inserter is loosely
based on the existing DfsInserter implementation, but is simpler since
we don't need to buffer blocks in memory before writing to storage.

An alternative for such applications would be to write out the loose
objects and then repack just those objects later. This operation is not
currently supported with the GC class, which always repacks existing
packs when compacting loose objects. This in turn requires more
CPU-intensive reachability checks and extra I/O to copy objects from old
packs to new packs.

So, the choice was between implementing a new variant of repack, or not
writing loose objects in the first place. The latter approach is likely
less code overall, and avoids unnecessary I/O at runtime.

The current implementation does not yet support newReader() for reading
back objects.

Change-Id: I2074418f4e65853b7113de5eaced3a6b037d1a17
2017-11-01 12:40:53 -04:00
Han-Wen NIenhuys dc24383b6b Revert "Throw BinaryBlobException from RawParseUtils#lineMap."
This reverts commit f2e64cd895.

The newly added throws clause breaks backward compatibility. 

Change-Id: Ifa76a1b95935e52640b81cd53c171eb17da175c2
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
2017-10-24 11:26:10 -04:00
Han-Wen Nienhuys f2e64cd895 Throw BinaryBlobException from RawParseUtils#lineMap.
This makes detection of binaries exact for ResolveMerger and
DiffFormatter: they will classify files as binary regardless of where
the '\0' occurs in the text.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Id4342a199628d9406bfa04af1b023c27a47d4014
2017-10-24 15:31:34 +02:00
Han-Wen Nienhuys ced658c445 Avoid loading and merging binary data in ResolveMerger
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Ide4b68872d426aa262142f224acf636c776b35d3
2017-10-24 15:07:04 +02:00
Han-Wen Nienhuys ea2a4e3abe Introduce RawText#load.
This method creates a RawText from a blob, but avoids reading the blob
if the start contains null bytes. This should reduce the amount of
garbage that Gerrit produces for changes with binaries.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Idd202d20251f2d1653e5f1ca374fe644c2cf205f
2017-10-24 14:49:10 +02:00
David Pursehouse f89101105e Merge branch 'stable-4.9'
* stable-4.9:
  Avoid bad rounding "1 year, 12 months" in date formatter
  Ensure that ~ in ssh config is replaced before Jsch sees it

Change-Id: If6ca55f9447aaea3d7c2d36c03520d5e6dd5193e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-10-23 12:08:59 +09:00
Michael Keppler e1a39cbbe7 Avoid bad rounding "1 year, 12 months" in date formatter
Round first, then calculate the labels. This avoids "x years, 12 months"
and instead produces "x+1 years".

One test case has been added for the original example the bug was found
with, and one assertion has been moved from an existing test case to the
new test case, since it also triggered the bug.

Bug: 525907
Change-Id: I3270af3850c4fb7bae9123a0a6582f93055c9780
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-10-23 11:33:55 +09:00
Shawn Pearce 7cd5d77ae3 dfs: Switch InMemoryRepository to DfsReftableDatabase
This ensure DfsReftableDatabase is tested by the same test suites that
use/test InMemoryRepository. It also simplifies the logic of
InMemoryRepository and brings its compatibility story closer to any
other DFS repository that uses reftables for its reference storage.

Change-Id: I881469fd77ed11a9239b477633510b8c482a19ca
Signed-off-by: Minh Thai <mthai@google.com>
Signed-off-by: Terry Parker <tparker@google.com>
2017-10-18 17:35:27 -07:00
Terry Parker 4b75d5223a Merge changes from topic 'reftable'
* changes:
  dfs: reftable backed DfsRefDatabase
  Support symbolic references in ReceiveCommand
2017-10-18 20:25:08 -04:00