Commit Graph

1890 Commits

Author SHA1 Message Date
Dave Borowitz edb6593ef0 ResolveMergerTest: Add tests for content merges
Change-Id: I2b85cc239e9b3090426c956cbf9af3deb4106a41
2017-04-04 17:27:15 -04:00
David Pursehouse 61a943e03d PushConnectionTest#limitCommandBytes: Increase maxCommandBytes
A higher limit is required to account for proper JGit version number
being sent in the UserAgent.

The version string "4.7.0.201704031717-r" is 20 characters, however
the strings used during development are shorter:

- When running from mvn, "4.7.0.qualifier" is used; 15 characters
- When running in Eclipse, "unknown" is used; 7 characters

Change-Id: I9aca2f71389a42fedce305e9078db016869c3d1a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-04-04 21:37:48 +09:00
KB Sriram 4a985f5aa8 Make diff locations more consistent
DiffAlgorithms can return different edit locations for inserts or
deletes, if they can be "shifted" up or down repeating blocks of
lines. This causes the 3-way merge to apply both edits, resulting in
incorrectly removing or duplicating lines.

Augment an existing "tidy-up" stage in DiffAlgorithm to move all
shiftable edits (not just the last INSERT edit) to a consistent
location, and add test cases for previously incorrect merges.

Bug: 514095
Change-Id: I5fe150a2fc04e1cdb012d22609d86df16dfb0b7e
Signed-off-by: KB Sriram <kbsriram@google.com>
2017-04-03 16:45:13 -07:00
Matthias Sohn 11a12ceb0b Prepare 4.5.2-SNAPSHOT builds
Change-Id: I8485de1f3f63dc9ec445b8fb08093ca144aedc59
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-03-25 01:21:58 +01:00
David Pursehouse 5f902f07cc PullCommand: Add support for recursing into submodules
Add a new API method to set the recurse mode, and pass the mode into
the fetch command.

Extend the existing FetchCommandRecurseSubmodulesTest to also perform
the same tests for fetch. Rename the test class accordingly.

Change-Id: I12553af47774b4778f7011e1018bd575a7909bd0
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-03-24 00:02:45 +01:00
David Ostrovsky fa2851ce78 bazel: Consume hamcrest through transitive dependency
In I3ab958ce8 explicit dependency in lib/BUILD were defined and most
of the bazel build implementation was switched to using it. Switch
test.bzl test implementation to using explicit dependencies as well.

Change-Id: I4413d1a45addeeb2a980d07669fa034c2eebb3a4
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2017-03-23 06:47:49 +01:00
David Pursehouse 2d0ce094b4 Remove Buck build
Buck will be replaced with Bazel

Change-Id: I3cf07d7aaaa2a58bac34e16c50af5416693254ac
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-03-22 01:41:21 +01:00
David Ostrovsky 0f6ddb372b Add remaining parts of the bazel build
Add bazel build for ui and junit.http, and the test packages.

A number of different test labels are supported:

  api
  attributes
  dfs
  diff
  http
  lfs
  lfs-server
  nls
  notes
  pack
  patch
  pgm
  reftree
  revplot
  revwalk
  storage
  submodule
  symlinks
  transport
  treewalk
  util

To run all tests:

  bazel test //...

To run specific tests, using labels:

  bazel test --test_tag_filters=api,dfs,revplot,treewalk //...

Change-Id: Ic41b05a79d855212e67b1b4707e9c6b4dc9ea70d
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Signed-off-by: Jonathan Nieder <jrn@google.com>
2017-03-21 09:24:12 +01:00
David Ostrovsky cee9d444e9 Delegate crypto algorithm creation to InsecureCipherFactory
This is a preparation change to Bazel build implementation. Error
Prone rejects the code with variable crypto algorithm as insecure
see: [1].

[1] http://errorprone.info/bugpattern/InsecureCryptoUsage

Change-Id: I92db70a7da454bc364597a995e8be5dccc2d6427
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2017-03-21 09:23:49 +01:00
Matthias Sohn a9a84b7235 JGit v4.5.1.201703201650-r
Change-Id: I88de7c9f52abbc4921a82208ed74d22aa19fb3cd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-03-20 21:44:47 +01:00
David Pursehouse 5e5fbb6035 Fix test configuration to run RacyGitTests, and fix testRacyGitDetection
With the filename suffix "Tests", the module was not included in tests
when building with Maven, and without the @Test annotations the tests
didn't get executed under Eclipse or buck test.

testRacyGitDetection was failing because the index file did not exist.

Add the missing configuration, the missing annotations, and add a call
to reset() in testRacyGitDetection to force creation of the index file.

Change-Id: I29dd8f89c36fef4ab40bedce7f4a26bd9b2390e4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-03-19 10:56:06 +09:00
David Ostrovsky 9b4f1913fb RevFlagSetTest: Fix compilation error flagged by error prone
This fixes error flagged by error prone:

Java compilation in rule '//org.eclipse.jgit.test:jgit' failed: Worker
process sent response with exit code: 1.
org.eclipse.jgit.test/tst/org/eclipse/jgit/revwalk/RevFlagSetTest.java:149:
error: [CollectionIncompatibleType] Argument '"bob"' should not be
passed to this method; its type String is not compatible with its
collection's type argument RevFlag
		assertFalse(set.contains("bob"));

Change-Id: I4a971ce92fee55e28b2ab0c7b716ac20fa9c6709
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2017-03-18 16:47:16 +01:00
David Pursehouse 2fe1a3abbe FetchCommand: Fix detection of submodule recursion mode
The submodule.name.fetchRecurseSubmodules value was being read from the
configuration of the submodule, but it should be read from the config
of the parent repository.

Also, the fetch.recurseSubmodules value from the parent repository's
configuration was not being considered at all.

Fix both of these and add tests. Now the precedence of the recurse mode
is determined as follows:

 1. Value passed to the API
 2. Value configured in submodule.name.fetchRecurseSubmodules
 3. Value configured in fetch.recurseSubmodules
 4. Default to "on demand"

Change-Id: Ic23b7c40b5f39135fb3fd754c597dd4bcc94240c
2017-03-10 13:17:39 +09:00
Matthias Sohn 79f85d1cf2 Prepare 4.6.2-SNAPSHOT builds
Change-Id: I8835f79145e6a989787d47322c3d8cb9baf0624a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-03-07 20:49:24 +01:00
Matthias Sohn 258dc5a715 JGit v4.6.1.201703071140-r
Change-Id: I842dc95313e5b47b0b7ec983c4a0a91915ed4183
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-03-07 17:40:32 +01:00
David Pursehouse 503d59044f FetchCommand: Add basic support for recursing into submodules
Extend FetchCommand to expose a new method, setRecurseSubmodules(mode),
which allows to set the mode to ON, OFF or ON_DEMAND.

After fetching a repository, its submodules are recursively fetched:

- When the mode is YES, submodules are always fetched.

- When the mode is NO, submodules are not fetched.

- When the mode is ON_DEMAND, submodules are only fetched when the
  parent repository receives an update of the submodule and the new
  revision is not already in the submodule.

The mode is determined in the following order of precedence:

- Value specified in the API call using setRecurseSubmodules.

- Value specified in the repository's config under the key
  submodule.name.fetchRecurseSubmodules

- Defaults to ON_DEMAND if neither of the previous is set.

Extend FetchResult to recursively include results for submodules, as
a map of the submodule path to an instance of FetchResult.

Test setup is based on testCloneRepositoryWithNestedSubmodules.

Change-Id: Ibc841683763307cb76e78e142e0da5b11b1add2a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-03-04 09:31:16 +09:00
Thomas Wolf 1f3e74ed9f Make Repository.normalizeBranchName less strict
This operation was added recently with the goal to provide some
way to auto-correct invalid user input, or to provide a correction
suggestion to the user -- EGit uses it now that way. But the initial
implementation was very restrictive; it removed all non-ASCII
characters and even slashes.

Understandably end users were not happy with that. Git has no such
restriction to ASCII-only; nor does JGit. Branch names should be
meaningful to the end user, and if a user-supplied branch name is
invalid for technical reasons, a "normalized" name should still
be meaningful to the user.

Rewrite to attempt a minimal fix such that the result will pass
isValidRefName.

* Replace all Unicode whitespace by underscore.
* Replace troublesome special characters by dash.
* Collapse sequences of underscores, dots, and dashes.
* Remove underscores, dots, and dashes following slashes, and
  collapse sequences of slashes.
* Strip leading and trailing sequences of slashes, dots, dashes,
  and underscores.
* Avoid the ".lock" extension.
* Avoid the Windows reserved device names.
* If input name is null return an empty String so callers don't need to
check for null.

This still allows branch names with single slashes as separators
between components, avoids some pitfalls that isValidRefName() tests
for, and leaves other character untouched and thus allows non-ASCII
branch names.

Also move the function from the bottom of the file up to where
isValidRefName is implemented.

Bug: 512508
Change-Id: Ia0576d9b2489162208c05e51c6d54e9f0c88c3a7
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-03-04 00:23:42 +01:00
Jonathan Nieder 45f62576de Merge "SHA-1: collision detection support" 2017-03-02 13:26:45 -05:00
Shawn Pearce 83ad74b6b9 SHA-1: collision detection support
Update SHA1 class to include a Java port of sha1dc[1]'s ubc_check,
which can detect the attack pattern used by the SHAttered[2] authors.

Given the shattered example files that have the same SHA-1, this
modified implementation can identify there is risk of collision given
only one file in the pair:

  $ jgit ...
  [main] WARN org.eclipse.jgit.util.sha1.SHA1 - SHA-1 collision 38762cf7f55934b34d179ae6a4c80cadccbb7f0a

When JGit detects probability of a collision the SHA1 class now warns
on the logger, reporting the object's SHA-1 hash, and then throws a
Sha1CollisionException to the caller.

From the paper[3] by Marc Stevens, the probability of a false positive
identification of a collision is about 14 * 2^(-160), sufficiently low
enough for any detected collision to likely be a real collision.

git-core[4] may adopt sha1dc before the system migrates to an entirely
new hash function.  This commit enables JGit to remain compatible with
that move to sha1dc, and help protect users by warning if similar
attacks as SHAttered are identified.

Performance declined about 8% (detection off), now:

  MessageDigest        238.41 MiB/s
  MessageDigest        244.52 MiB/s
  MessageDigest        244.06 MiB/s
  MessageDigest        242.58 MiB/s

  SHA1                 216.77 MiB/s (was ~240.83 MiB/s)
  SHA1                 220.98 MiB/s
  SHA1                 221.76 MiB/s
  SHA1                 221.34 MiB/s

This decline in throughput is attributed to the step loop unrolling in
compress(), which was necessary to easily fit the UbcCheck logic into
the hash function.  Using helper functions s1-s4 reduces the code
explosion, providing acceptable throughput.

With detection enabled (default):

  SHA1 detectCollision 180.12 MiB/s
  SHA1 detectCollision 181.59 MiB/s
  SHA1 detectCollision 181.64 MiB/s
  SHA1 detectCollision 182.24 MiB/s

  sha1dc (native C)   ~206.28 MiB/s
  sha1dc (native C)   ~204.47 MiB/s
  sha1dc (native C)   ~203.74 MiB/s

Average time across 100,000 calls to hash 4100 bytes (such as a commit
or tree) for the various algorithms available to JGit also shows SHA1
is slower than MessageDigest, but by an acceptable margin:

  MessageDigest        17 usec
  SHA1                 18 usec
  SHA1 detectCollision 22 usec

Time to index-pack for git.git (217982 objects, 69 MiB) has increased:

  MessageDigest   SHA1 w/ detectCollision
  -------------   -----------------------
         20.12s   25.25s
         19.87s   25.48s
         20.04s   25.26s

    avg  20.01s   25.33s    +26%

Being implemented in Java with these additional safety checks is
clearly a penalty, but throughput is still acceptable given the
increased security against object name collisions.

[1] https://github.com/cr-marcstevens/sha1collisiondetection
[2] https://shattered.it/
[3] https://marc-stevens.nl/research/papers/C13-S.pdf
[4] https://public-inbox.org/git/20170223230621.43anex65ndoqbgnf@sigill.intra.peff.net/

Change-Id: I9fe4c6d8fc5e5a661af72cd3246c9e67b1b9fee6
2017-02-28 16:38:43 -08:00
Magnus Vigerlöf 2a5d20c138 Correct the boolean logic for filtering paths
The TreeWalk filtering classes need to support the three different
meanings of the return value the path comparison generates.
A new path comparison method (isPathMatch) is created with
three distinct return values (isPathPrefix use value '0' to
encode two of these) which will makes it possible for the logical
operators (especially NOT) to aggregate a correct verdict.

A filter like: AND(Path("path"), NOT(Path("path/to/other")))
Should filter out 'path/to/other/file', but not 'path/to/my/file'.

The path-limiting feature when testing path/to/my/file, would
result to run test for the following paths:

    path
    path/to
    path/to/my
    path/to/my/file

isPathPrefix('path/to/other') will return '0' for the first two
and since there is no way for NOT to distinguish between an exact
match and a match indicating that the tested path is a 'parent',
it will incorrectly return false and thus remove everything below
'path' immediately.
isPathMatch has a distinguished value for 'parent' matches that
will be preserved through the logic operators and should not
cause an over-eager removal of paths.

The functionality of isPathPrefix is required by other parts
and is untouched.

Unit tests are included to ensure that the logical functionality
is correct and can be preserved.

Change-Id: Ice2ca9406f09f1b179569e99b86a0e5d77baa20d
Signed-off-by: Magnus Vigerlöf <magnus.vigerlof@gmail.com>
2017-02-28 23:56:33 +01:00
Shawn Pearce 1bf7d3f290 SHA1: support reset() and reuse instances
Allow SHA1 instances to be reused to compute another hash value, and
resume caching them in ObjectInserter and PackParser.  This shaves a
small amount of running time off parsing git.git's pack file:

  before   after
  ------   ------
  25.25s   25.55s
  25.48s   25.06s
  25.26s   24.94s

Almost noise (small difference), but recycling the instances reduces
some stress on the memory allocator finding two 80 word message block
arrays needed for hashing and collision detection.

Change-Id: I4af88a720e81460293bc5c5d1d3db1a831e7e228
2017-02-26 15:26:53 -08:00
Shawn Pearce 982f5d1bf1 Pure Java SHA-1
This implementation is derived straight from the description written
in RFC 3174.  On Mac OS X with Java 1.8.0_91 it offers similar
throughput as MessageDigest SHA-1:

  system   239.75 MiB/s
  system   244.71 MiB/s
  system   245.00 MiB/s
  system   244.92 MiB/s

  sha1     234.08 MiB/s
  sha1     244.50 MiB/s
  sha1     242.99 MiB/s
  sha1     241.73 MiB/s

This is the fastest implementation I could come up with.  Common SHA-1
implementation tricks such as unrolling loops creates a method too
large for the JIT to effectively optimize, resulting in lower overall
hashing throughput. Using a preprocessor to perform the register
renaming of A-E also didn't help, as again the method was too large
for the JIT to effectively optimize.

Fortunately the fastest version is a naive, straight-forward
implementation very close to the description in RFC 3174.

Change-Id: I228b05c4a294ca2ad51386cf0e47978c68e1aa42
2017-02-26 11:16:19 -08:00
David Pursehouse 3b4448637f Enable and fix warnings about redundant specification of type arguments
Since the introduction of generic type parameter inference in Java 7,
it's not necessary to explicitly specify the type of generic parameters.

Enable the warning in Eclipse, and fix all occurrences.

Change-Id: I9158caf1beca5e4980b6240ac401f3868520aad0
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-02-20 22:47:23 +01:00
David Pursehouse fceac7e44d Add some more missing @Override annotations
Change-Id: Ic13160920b986edde87c928c473240cc9c034f50
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-02-20 11:32:22 +09:00
David Pursehouse 7ac182f4e4 Enable and fix 'Should be tagged with @Override' warning
Set missingOverrideAnnotation=warning in Eclipse compiler preferences
which enables the warning:

  The method <method> of type <type> should be tagged with @Override
  since it actually overrides a superclass method

Justification for this warning is described in:

  http://stackoverflow.com/a/94411/381622

Enabling this causes in excess of 1000 warnings across the entire
code-base. They are very easy to fix automatically with Eclipse's
"Quick Fix" tool.

Fix all of them except 2 which cause compilation failure when the
project is built with mvn; add TODO comments on those for further
investigation.

Change-Id: I5772061041fd361fe93137fd8b0ad356e748a29c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-02-19 20:05:08 -04:00
Naoki Takezoe 1448ec37f9 Set commit time to ZipArchiveEntry
Archived zip files for a same commit have different MD5 hash because
mdate and mdate in the header of zip entries are not specified. In
this case, Commons Compress sets an archived time.

In the original git implementation, it's set a commit time:
e2b2d6a172/archive.c (L378)

By this fix, archive command sets the commit time to ZipArchiveEntry
when RevCommit is given as an archiving target.

Change-Id: I30dd8710e910cdf42d57742f8709e9803930a123
Signed-off-by: Naoki Takezoe <takezoe@gmail.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-02-18 10:47:27 +01:00
David Turner d3962fef6b GC: don't loosen doomed objects
If the pruneexpire config is set to "now", then any unreferenced loose
objects are immediately eligible for gc.  So there is no need to
actually write the loose objects.

Users who run hosting services which sometimes accept large, entirely
garbage packs might set the following configurations:

gc.pruneExpire = now
gc.prunePackExpire = 2.weeks

Then garbage objects will be kept around in packs, but after two weeks
the packs themselves will get deleted.

For client-side users of jgit, the default settings will loosen
garbage objects, and, after an hour, delete the old packs in which
they resided.

Change-Id: I8f686ac60b40181b1ee92ac6c313c3f33b55c44c
Signed-off-by: David Turner <dturner@twosigma.com>
2017-02-17 11:26:09 -05:00
Shawn Pearce 0bff481d45 Limit receive commands
Place a configurable upper bound on the amount of command data
received from clients during `git push`.  The limit is applied to the
encoded wire protocol format, not the JGit in-memory representation.
This allows clients to flexibly use the limit; shorter reference names
allow for more commands, longer reference names permit fewer commands
per batch.

Based on data gathered from many repositories at $DAY_JOB, the average
reference name is well under 200 bytes when encoded in UTF-8 (the wire
encoding).  The new 3 MiB default receive.maxCommandBytes allows about
11,155 references in a single `git push` invocation.  A Gerrit Code
Review system with six-digit change numbers could still encode 29,399
references in the 3 MiB maxCommandBytes limit.

Change-Id: I84317d396d25ab1b46820e43ae2b73943646032c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-02-11 00:20:36 +01:00
David Pursehouse 158d3722d3 IndexDiffSubmoduleTest: Fix negative use count
submoduleStandalone is created by createWorkRepository() which adds
the created repository to the set of repositories to be closed in
the test teardown. It is therefore not necessary to explicitly close
it.

Change-Id: Ib6f525b644fdeaaf1934df39cc2d3583a0d883dc
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-02-10 19:48:52 +09:00
David Pursehouse 51239129b3 FetchCommandTest: Don't declare specific exceptions in test methods
Change-Id: Ie0f8a0f7a9c2c383be6ae8265353daac7f5a89fa
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-02-09 15:10:15 +09:00
David Pursehouse 16dc88fe10 PushCommandTest: Remove unused variables to prevent errors in Eclipse
Change-Id: Ie656b18fb151bf1e3c2dcc0438a77e32102991c2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-02-09 15:10:15 +09:00
Shawn Pearce 8fce17a995 Merge "push: support per-ref force-with-lease" 2017-02-08 22:27:06 -05:00
David Turner 46d35a8502 push: support per-ref force-with-lease
When rebasing, force-pushing has a race condition: someone else might
have pushed a commit since the one you just rewrote. The force-with-lease
option prevents this by ensuring that the ref's old value is the one
that you expected.

Change-Id: I97ca9f8395396c76332bdd07c486e60549ca4401
Signed-off-by: David Turner <dturner@twosigma.com>
2017-02-08 19:42:33 -05:00
Shawn Pearce d67b183537 Prefer smaller GC files during DFS garbage collection
In 8ac65d33ed PackWriter changed its
behavior to always prefer the last object representation presented
to it by the ObjectReuseAsIs implementation. This was a fix to avoid
delta chain cycles.

Unfortunately it can lead to suboptimal compression when concurrent
GCs are run on the same repository. One case is automatic GC running
(with default settings) in parallel to a manual GC that has disabled
delta reuse in order to generate new smaller deltas for the entire
history of the repository.

Running GC with no-reuse generally requires more CPU time, which
also translates to a longer running time.  This can lead to a race
where the automatic GC completes before the no-reuse GC, leaving
the repository in a state such as:

  no-reuse GC:   size 1 GiB, mtime = 18:45
  auto GC:       size 8 GiB, mtime = 17:30

With the default sort ordering, the smaller no-reuse GC pack is
sorted earlier in the pack list, due to its more recent mtime.

During object reuse in a future GC, these smaller representations
are considered first by PackWriter, but are all discarded when the
auto GC file from 17:30 is examined second (due to its older mtime).

Work around this in two ways.

Well formed DFS repositories should have at most 1 GC pack. If
2 or more GC packs exist, break the sorting tie by selecting the
smaller file earlier in the pack list. This allows all normal read
code paths to favor the smaller file, which places less pressure
on the DfsBlockCache. If any GC race happens, readers serving clone
requests will prefer the file that is smaller.

During object reuse, flip this ordering so that the smaller file is
last. This allows PackWriter to see smaller deltas last, replacing
larger representations that were previously considered from other
pack files.

Change-Id: I0b7dc8bb9711c82abd6bd16643f518cfccc6d31a
2017-02-08 14:37:12 -08:00
Shawn Pearce 61d4922928 Fix missing deltas near type boundaries
Delta search was discarding discovered deltas if an object appeared
near a type boundary in the delta search window. This has caused JGit
to produce larger pack files than other implementations of the packing
algorithm.

Delta search works by pushing prior objects into a search window, an
ordered list of objects to attempt to delta compress the next object
against. (The window size is bounded, avoiding O(N^2) behavior.)

For implementation reasons multiple object types can appear in the
input list, and the window. PackWriter commonly passes both trees and
blobs in the input list handed to the DeltaWindow algorithm. The pack
file format requires an object to only delta compress against the same
type, so the DeltaWindow algorithm must stop doing comparisions if a
blob would be compared to a tree.

Because the input list is sorted by object type and the window is
recently considered prior objects, once a wrong type is discovered in
the window the search algorithm stops and uses the current result.

Unfortunately the termination condition was discarding any found
delta by setting deltaBase and deltaBuf to null when it was trying
to break the window search.

When this bug occurs, the state of the DeltaWindow looks like this:

                                 current
                                  |
                                 \ /
  input list:  tree0 tree1 blob1 blob2

  window:      blob1 tree1 tree0
                / \
                 |
              res.prev

As the loop iterates to the right across the window, it first finds
that blob1 is a suitable delta base for blob2, and temporarily holds
this in the bestDelta/deltaBuf fields. It then considers tree1, but
tree1 has the wrong type (blob != tree), so the window loop must give
up and fall through the remaining code.

Moving the condition up and discarding the window contents allows
the bestDelta/deltaBuf to be kept, letting the final file delta
compress blob1 against blob0.

The impact of this bug (and its fix) on real world repositories is
likely minimal. The boundary from blob to tree happens approximately
once in the search, as the input list is sorted by type. Only the
first window size worth of blobs (e.g. 10 or 250) were failing to
produce a delta in the final file.

This bug fix does produce significantly different results for small
test repositories created in the unit test suite, such as when a pack
may contains 6 objects (2 commits, 2 trees, 2 blobs).  Packing test
cases can now better sample different output pack file sizes depending
on delta compression and object reuse flags in PackConfig.

Change-Id: Ibec09398d0305d4dbc0c66fce1daaf38eb71148f
2017-02-08 14:36:24 -08:00
Shawn Pearce 12c8462602 Merge "Reintroduce garbage pack coalescing when ttl > 0." 2017-02-08 00:23:40 -05:00
Thirumala Reddy Mutchukota 006f4d4d29 Reintroduce garbage pack coalescing when ttl > 0.
Disabling the garbage pack coalescing when garbageTtl > 0 can result in
lot of garbage packs if they are created within the garbageTtl time.

To avoid a large number of garbage packs, re-introducing garbage pack
coalescing for the packs that are created within a single calendar day
when the garbageTtl is more than one day or one third of the garbageTtl.

Change-Id: If969716aeb55fb4fd0ff71d75f41a07638cd5a69
Signed-off-by: Thirumala Reddy Mutchukota <thirumala@google.com>
2017-02-07 20:34:31 -08:00
Matthias Sohn f8d232213c Branch normalizer should not normalize already valid branch names
Change-Id: Ib746655e32a37c4ad323f1d12ac0817de8fa56cf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-02-07 00:24:39 +01:00
Hongkai Liu a33663fd4e Detect stale-file-handle error in causal chain
Cover the case where the exception is wrapped up as a
cause, e.g., PackIndex#open(File).

Change-Id: I0df5b1e9c2ff886bdd84dee3658b6a50866699d1
Signed-off-by: Hongkai Liu <hongkai.liu@ericsson.com>
2017-01-30 22:36:59 -04:00
David Pursehouse 62411453f1 Merge branch 'stable-4.6'
* stable-4.6:
  Clean up orphan files in GC

Change-Id: I4fb6b4cd03d032535a9c04ede784bea880b4536b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-01-31 09:31:10 +09:00
David Pursehouse 25ab5b4d9b Merge "Don't rely on default locale when using toUpperCase() and toLowerCase()" 2017-01-30 07:32:32 -05:00
David Pursehouse c64412e1f9 Merge FileTreeIteratorJava7Test into FileTreeIteratorTest
JGit now requires Java 8, so it is no longer necessary to have a
separate class for Java 7 specific tests. Remove it and merge its
tests into the existing FileTreeIteratorTest.

FileTreeIteratorTest has an @Before annotated method that sets up
some files in the git, which breaks the tests which have assumptions
on the file names. Add adjustments.

Change-Id: I14f88d8e079e1677c8dfbc1fcbf4444ea8265365
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-01-30 13:23:52 +01:00
Hongkai Liu b198f77c1e Rename FileUtilTest to FileUtilsTest and merge in FileUtils7Test
Rename the test class to match the name of the class under test.

JGit now requires Java 8 so it is no longer necessary to have a
separate class (FileUtils7Test) for Java 7 tests. Merge those into
FileUtilsTest.

Change-Id: I39dd7e76a2e4ce97319c7d52261b0a1546879788
Signed-off-by: Hongkai Liu <hongkai.liu@ericsson.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-01-30 02:42:56 -04:00
Hongkai Liu 8fd500e20c Clean up orphan files in GC
An orphan file is either a bitmap or an idx file in pack folder,
and its corresponding pack file is missing.

Change-Id: I3c4cb1f7aa99dd7b398bdb8d513f528d7761edff
Signed-off-by: Hongkai Liu <hongkai.liu@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-01-30 00:55:36 +01:00
Matthias Sohn 53ad437382 RepositoryCacheTest: avoid to close already closed repository
The tearDown() of the superclass closed the repository once more which
led to a negative use count warning logged by Repository.close().

Change-Id: I331f85a540c68264a53456276c32f72b79113d61
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-01-28 21:19:55 +01:00
David Pursehouse ac6353e9e5 RefUpdateTest: Don't call createBareRepository in try-with-resource
createBareRepository adds the created repo to the list of repos to be
closed in the superclass's teardown. Wrapping it in try-with-resource
causes it to be closed too many times, resulting in a corrupt use
count.

Change-Id: I4c70630bf6008544324dda453deb141f4f89472c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-01-28 19:16:48 +01:00
David Pursehouse 25b14084c9 ReceivePackAdvertiseRefsHookTest: Don't close repositories in teardown
The repositories get added to the "toClose" set by createBareRepository,
and are then closed in the superclass's tearDown method.

Explicitly closing them in this test class's teardown causes the use
count to go negative when subsequently closed again by the superclass.

Change-Id: Idcbb16b4cf4bf0640d7e4ac15d1926d8a27c1251
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-01-28 17:46:28 +01:00
David Pursehouse 883856110c RepoCommandTest: Don't wrap create{Bare,Work}Directory in t-w-r
These methods add the created Repository into "toClose", and they are
then closed by LocalDiskRepositoryTestCase's tearDown method.

Calling them in try-with-resource causes them to first be closed in
the test method, and then again in tearDown, which results in the use
count going negative and a log message on the console.

While this is not a serious problem, having so many false positives
in the logs will potentially drown out real cases of Repository being
closed too many times.

Change-Id: Ib374445e101dc11cb840957b8b19ee1caf777392
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-01-28 17:46:27 +01:00
David Pursehouse a9a3af4af7 GitConstructionTest: Remove unnecessary calls to Repository.close()
The repositories are already closed in the superclass teardown due
to being added to the "toClose" set.

Change-Id: I768ba8a02fc585907687caf37e2e283434016c04
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-01-28 17:46:27 +01:00
Matthias Sohn a4feeb0194 Don't rely on default locale when using toUpperCase() and toLowerCase()
Otherwise these methods may produce unexpected results if used for
strings that are intended to be interpreted locale independently.
Examples are programming language identifiers, protocol keys, and HTML
tags. For instance, "TITLE".toLowerCase() in a Turkish locale returns
"t\u0131tle", where '\u0131' is the LATIN SMALL LETTER DOTLESS I
character.

See
https://docs.oracle.com/javase/8/docs/api/java/lang/String.html#toLowerCase--
http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html

Bug: 511238
Change-Id: Id8d8f37d84d62239c918b81f8d883ed798d87656
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-01-28 15:06:15 +01:00
Thirumala Reddy Mutchukota c9f55032a2 Record the estimated size of the pack files.
The Compacter and Garbage Collector will record the estimated size of
the newly going to be created compact, gc or garbage packs. This
information can be used by the clients to better make a call on how to
actually store the pack based on the approximated expected size.

Added a new protected method DfsObjDatabase.newPack(PackSource
packSource, long estimatedPackSize), so that the clients can override
this method to make use of the estimatedPackSize while creating a new
PackDescription object. The default implementation of this method is
equivalent to
newPack(packSource).setEstimatedPackSize(estimatedPackSize). I didn't
make it abstract because that would force all the existing sub classes
of DfsObjDatabase to implement this method. Due to this default
implementation, the estimatedPackSize is added to DfsPackDescription
using a setter instead of a constructor parameter (even though
constructor parameter would be a better choice as this value is set only
during the object creation).

Change-Id: Iade1122633ea774c2e842178a6a6cbb4a57b598b
Signed-off-by: Thirumala Reddy Mutchukota <thirumala@google.com>
2017-01-26 12:01:59 -08:00
David Turner 8bec98cec0 gc: loosen unreferenced objects
An unreferenced object might appear in a pack.  This could only happen
because it was previously referenced, and then later that reference
was removed.  When we gc, we copy the referenced objects into a new
pack, and delete the old pack.  This would remove the unreferenced
object.  Now we first create a loose object from any unreferenced
object in the doomed pack.  This kicks off the two-week grace period
for that object, after which it will be collected if it's not
referenced.

This matches the behavior of regular git.

Change-Id: I59539aca1d0d83622c41aa9bfbdd72fa868ee9fb
Signed-off-by: David Turner <dturner@twosigma.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
2017-01-24 14:22:45 -08:00
James Melvin 91132bb05e gc: Add options to preserve and prune old pack files
The new --preserve-oldpacks option moves old pack files into the
preserved subdirectory instead of deleting them after repacking.

The new --prune-preserved option prunes old pack files from the
preserved subdirectory after repacking, but before potentially
moving the latest old packfiles to this subdirectory.

These options are designed to prevent stale file handle exceptions
during git operations which can happen on users of NFS repos when
repacking is done on them. The strategy is to preserve old pack files
around until the next repack with the hopes that they will become
unreferenced by then and not cause any exceptions to running processes
when they are finally deleted (pruned).

Change-Id: If3f729f0d9ce920ee2c3e6acdde46f2068be61d2
Signed-off-by: James Melvin <jmelvin@codeaurora.org>
2017-01-19 11:00:18 +01:00
Wim Jongman b667c182cb Normalizer creating a valid branch name from a string
Generic normalization method for a possible invalid branch name.
The method compresses dividers between spaces, then replaces spaces
and non word characters with underscores.

This method is needed in preparation for subsequent EGit changes.

Bug: 509878
Change-Id: Ic0d12f098f90f912a45bcc5693d6accf751d4e58
Signed-off-by: Wim Jongman <wim.jongman@remainsoftware.com>
2017-01-18 22:05:28 +01:00
Christian Halstrick 8a46b60371 Merge "Fix StashApplyCommand for stashes containing untracked changes." 2017-01-16 03:45:00 -05:00
Thomas Wolf 46af7192a2 Fix StashApplyCommand for stashes containing untracked changes.
If there are untracked changes, apply only the untracked tree
after a successful merge. The merge tree from merging untracked
with HEAD would also contain files already reset before (changes
in tracked files) and try to reset those again,leading to false
checkout conflicts.

Bug: 505804
Change-Id: Iaced4d277623334d11e3d1cca5969590d7c5093e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-01-15 21:54:12 +01:00
Marc Strapetz 1c4b3f8c45 Fix possible InvalidObjectIdException in ObjectDirectory
ObjectDirectory.getShallowCommits should throw an IOException
instead of an InvalidArgumentException if invalid SHAs are present
in .git/shallow (as this file is usually edited by a human).

Change-Id: Ia3a39d38f7aec4282109c7698438f0795fbec905
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-01-15 15:05:51 +01:00
Shawn Pearce db77610256 Pack refs/tags/ with refs/heads/
This fixes a nasty performance issue for repositories that have many
objects referenced through refs/tags/, but not in refs/heads/.
Situations like this can arise when a project has made releases like
refs/tags/v1.0, and then decides to orphan history and start over for
version 2. The v1.0 objects are not reachable from master anymore,
but are still live due to the v1.0 tag.

When tags are packed in the GC_OTHER pack, bitmaps are not able to
cover the repository's contents. This may cause very slow counting
times during git clone, as the server must enumerate the ancient
history under refs/tags/ to respond to the client.

Clients by default always ask for all tags when asking for all heads
during clone. This has been true since git-core commit 8434c2f1afedb
(Apr 27 2008), when clone was converted to a builtin. Including tags
in the main GC pack should still allow servers to benefit from the
fast full pack reuse path when serving a clone to a client.

Change-Id: I22e29517b5bc6fa3d6b19a19f13bef0c68afdca3
2017-01-03 14:46:41 -08:00
Matthias Sohn fc6f02cdba [findBugs] Fix potential NPE in ObjectDirectoryTest
Change-Id: I0307d1e9bbc0bfc8172eed10f8e93788d6e2c8b4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-12-29 01:06:06 +01:00
Matthias Sohn a85979c8b2 [findBugs] Fix potential NPE in GcPackRefsTest
Change-Id: I273c06b9b2ce36c8232803cb7f02d7f846ed49e5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-12-29 01:04:25 +01:00
Matthias Sohn 48b772bec1 [findBugs] Fix potential NPE in DirCacheCheckoutTest
Change-Id: I842cbdd14bf56cb2bd599255fe90a5a3aea0d304
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-12-29 00:59:31 +01:00
James Melvin d980a3fa85 Fix keep pack filename
Previously it was looking for a keep file with the name of a pack file
(extenstion included) appended with a '.keep'. However, the keep file
name should be the pack file name  with a '.keep' extension

Change-Id: I9dc4c7c393ae20aefa0b9507df8df83610ce4d42
Signed-off-by: James Melvin <jmelvin@codeaurora.org>
2016-12-27 14:08:56 -07:00
Matthias Sohn 5fee071f6a Prepare 4.7.0-SNAPSHOT builds
Change-Id: I20754d13007e6591d36aae5766f3a9a82b24e120
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-12-27 01:45:50 +01:00
Matthias Sohn 3857c3168f Prepare 4.6.1-SNAPSHOT builds
Change-Id: I6b05a6f6c3f92365c272e1bdaf76093ca01f2d58
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-12-24 15:51:54 +01:00
Matthias Sohn 73a4325149 JGit v4.6.0.201612231935-r
Change-Id: Iaa88fe1b195dfe6be99a7b4cb064684e75563715
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-12-24 01:42:38 +01:00
Christian Halstrick 11d24e6844 Fix FileSnapshot.isModified
FileSnapshot.isModified may have reported a file to be clean although it
was actually dirty.

Imagine you have a FileSnapshot on file f. lastmodified and lastread are
both t0. Now time is t1 and you
1) modify the file
2) update the FileSnapshot of the file (lastModified=t1, lastRead=t1)
3) modify the file again
4) wait 3 seconds
5) ask the Filesnapshot whether the file is dirty or not. It erroneously
answered it's clean.

Any file which has been modified longer than 2.5 seconds ago was
reported to be clean. As the test shows that's not always correct.

The real-world problem fixed by this change is the following:
* A gerrit server using JGit to serve git repositories is processing
fetch requests while simultaneously a native git garbage collection
runs on the repo.
* At time t1 native git writes temporary files in the pack folder
setting the mtime of the pack folder to t1.
* A fetch request causes JGit to search for new packfiles and JGit
remembers this scan in a Filesnapshot on the packs folder. Since the gc
is not finished JGit doesn't see any new packfiles.
* The fetch is processed and the gc ends while the filesystem timer is
still t1. GC writes a new packfile and deletes the old packfile.
* 3 seconds later another request arrives. JGit does not yet know about
the new packfile but is also not rescanning the pack folder because it
cached that the last scan happened at time t1 and pack folder's mtime is
also t1. Now JGit will not be able to resolve any object contained in
this new pack. This behavior may be persistent if objects referenced by
the ref/meta/config branch are affected so gerrit can't read permissions
stored in the refs/meta/config branch anymore and will not allow any
pushes anymore. The pack folder will not change its mtime and therefore
no rescan will take place.

Change-Id: I3efd0ccffeb97b01207dc3e7a6b85c6b06928fad
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-12-13 11:28:12 +01:00
Christian Halstrick 930cd43553 Fix merge-base calculation
Fix JGits merge-base calculation in case of inconsistent commit times.
JGit was potentially failing to compute correct merge-bases when the
commit times where inconsistent (a parent commit was younger than a
child commit). The code in MergeBaseGenerator was aware of the fact that
sometimes the discovery of a merge base x can occur after the parents of
x have been seen (see comment in #carryOntoOne()). But in the light of
inconsistent commit times it was possible that these parents of a
merge-base have already been returned as a merge-base.

This commit fixes the bug by buffering all commits generated by
MergeBaseGenerator. It is expected that this buffer will be small
because the number of merge-bases will be small. Additionally a new
flag is used to mark the ancestors of merge-bases. This allows to filter
out the unwanted commits.

Bug: 507584
Change-Id: I9cc140b784c3231b972bd2c3de61a789365237ab
2016-11-28 09:38:19 +01:00
Dave Borowitz 5bb434e01f Update JavaEWAH to 1.1.6
Use Oxygen M3 Orbit repository which provides the bundles built using
the new orbit-recipe based build.

CQ: 11658
Change-Id: I7f3dcc966732b32830c75d5daa55383bd028d182
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-11-17 00:26:44 +01:00
Shawn Pearce 8208da2f59 Deprecate SafeBufferedOutputStream
Java 8 fixed the silent flush during close issue by
FilterOutputStream (base class of BufferedOutputStream)
using try-with-resources to close the stream, getting a
behavior matching what JGit's SafeBufferedOutputStream
was doing:

  try {
    flush();
  } finally {
    out.close();
  }

With Java 8 as the minimum required version to run JGit
it is no longer necessary to override close() or have
this class. Deprecate the class, and use the JRE's version
of close.

Change-Id: Ic0584c140010278dbe4062df2e71be5df9a797b3
2016-11-14 15:33:54 -08:00
David Pursehouse a45cfee7a3 Organize imports
Change-Id: I7c545d06b1bced678c020fab9af1382bc4416b6e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-11-14 13:25:20 -08:00
Shawn Pearce 1c70dd6d21 Add {get,set}GitwebDescription to Repository
This method pair allows the caller to read and modify the description
file that is traditionally used by gitweb and cgit when rendering a
repository on the web.

Gerrit Code Review has offered this feature for years as part of
its GitRepositoryManager interface, but its fundamentally a feature
of JGit and its Repository abstraction.

git-core typically initializes a repository with a default value
inside the description file. During getDescription() this string
is converted to null as it is never a useful description.

Change-Id: I0a457026c74e9c73ea27e6f070d5fbaca3439be5
2016-11-14 11:14:35 -08:00
Philipp Marx 8adbfe4da6 Check that DfsBlockCache#blockSize is a power of 2
In case a value is used which isn’t a power of 2 there will be a high
chance of java.lang.ArrayIndexOutBoundsException and
org.eclipse.jgit.errors.CorruptObjectException due to a mismatching
assumption for the DfsBlockCache#blockSizeShift parameter.

Change-Id: Ib348b3704edf10b5f93a3ffab4fa6f09cbbae231
Signed-off-by: Philipp Marx <smigfu@googlemail.com>
2016-11-11 10:43:09 +01:00
Matthias Sohn f8ac03459a Fix loop in auto gc
* GC.tooManyLooseObjects() always responded true since the loop missed
to advance the iterator so it always incremented until the threshold was
exceeded.
* Also fix loop exit criterion which was off by 1.
* Add some tests.

Change-Id: I70976dfaa026efbcf3c46bd45941f37277a18e04
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-11-07 22:31:10 +01:00
Kevin Corcoran fa0a93119c Make streamFileThreshold configurable
Previously, the streamFileThreshold, the threshold at which a file
would be streamed rather than loaded entirely into memory, was only
configurable on a global basis.

This commit makes this threshold configurable on a per-loader basis.

Bug: 490404
Change-Id: I492c18c3155dbf56eedda9044a61d76120fd75f9
Signed-off-by: Kevin Corcoran <kevin.corcoran@puppetlabs.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-10-24 14:00:02 +09:00
David Turner e346873511 TreeFormatter: disallow empty filenames in trees
Git barfs on these (and they don't make any sense), so we certainly
shouldn't write them.

Change-Id: I3faf8554a05f0fd147be2e63fbe55987d3f88099
Signed-off-by: David Turner <dturner@twosigma.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-10-19 22:24:24 +09:00
David Pursehouse a5dde985a0 DiffCommandTest: Don't call toString on String instances
Change-Id: Ib308b3498593d595b3d8741a9b2d241bbc7441c3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-10-19 15:09:37 +09:00
David Pursehouse 9a7d28019a FileNameMatcherTest: Use Character.valueOf rather than new Character
Change-Id: I9d6e20a258d34ae1d2700fbe8e6c6e3b0ba94424
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-10-19 15:09:37 +09:00
David Pursehouse c0433f4fb7 Use valueOf rather than constructor for Integer and Boolean
Change-Id: I1c65b2e40ba6ec5860903b11b4631e014f3dc5ce
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-10-18 14:15:53 +09:00
David Pursehouse e0d1cfb5ad Upgrade buck to 7b7817c48f30687781040b2b82ac9218d5c4eaa4
Upgrade to match the version used on Gerrit's master branch.

Requires a couple of modifications to make the tests work:

- Remove source_under_test parameters from java_test calls.

- Add vm_args with explicit setting of tmpdir location for http
  tests. This is needed due to upstream changes in temporary
  directory handling [1].

[1] https://github.com/facebook/buck/issues/946

Change-Id: I5d5dd5edc335d44b118e8587f69ba89b83fc7fbb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-10-18 01:06:28 +02:00
Matthias Sohn ba7ba7a816 Merge branch 'stable-4.5'
* stable-4.5:
  Unconditionally close repositories in RepositoryCache.clear()
  Fix eviction of repositories with negative usage count

Adapt to parameter removed from
RepositoryCache.unregisterAndCloseRepository().

Change-Id: I7087667056ced401a3b3a027977f2715cd77a1c5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-10-14 00:24:00 +02:00
Hugo Arès 5a1e147eaa Fix eviction of repositories with negative usage count
If the repository close method was called twice (or more) for one open,
the usage count became negative and the repository was never be evicted
from the cache because the method checking if repository is expired was
not considering negative usage count.

Change-Id: I18a80c415c54c37d1b9def2b311ff2d0afa455ca
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
2016-10-12 07:07:03 -04:00
Thomas Wolf 26832a00e4 Fix symlink content comparison on MacOS in tree walk
Symlinks on MacOS are written as UTF-8 NFD, but
readSymbolicLink().toString() converts to NFC with potentially fewer
bytes. May occur in particular if the link target has non-ASCII
characters for which the NFC and NFD encodings differ. This may lead
to an EOFException: Short read of block.

This causes all kinds of weird effects in EGit, ranging from failing
rebases (which report the exception to the user) to EGit decorations in
the navigator silently disappearing (and never coming back).

* Rename readContentAsNormalizedString() to readSymlinkTarget() as it's
  called only for symlinks. Also make it protected.
* Fix by allowing the read to succeed even if less than the expected
  number of bytes are returned by the entry's input stream.
* Override in FileTreeIterator to use fs.readSymlink() directly.

Includes a new MacOS-only test.

Change-Id: I264c5972d67b1cbb1ed690580f5706e671b9affd
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-10-11 00:20:23 +02:00
Christian Halstrick 83e43f7960 Fix CheckoutCommand to return updated files even on NONDELETED status
CheckoutCommand was not returning updated and removed files in case of
an overall status of NONDELETED. That's status which occurs especially
on the Windows platform when Checkout wanted to delete files but the
filesystem doesn't allow this. The situation is more seldom on linux/mac
because open filehandles don't stop a deletion attempt and checkout
succeeds more often.

Change-Id: I4828008e58c09bd8f9edaf0f7eda0a79c629fb57
2016-09-27 15:58:24 +02:00
Matthias Sohn 8ed16fa100 Merge branch 'stable-4.5'
* stable-4.5:
  Fix carrying over flags during a RevWalk

Change-Id: Ibf4573c5664271dfa7a6ecc3ede6eaad749f89d8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-24 10:51:14 +02:00
Shawn Pearce 6257e27d49 Merge "Fix carrying over flags during a RevWalk" into stable-4.5 2016-09-24 01:16:49 -04:00
Christian Halstrick dd585b779f Fix carrying over flags during a RevWalk
There was a bug when carrying over flags from a merge commit to its
non-first parents. The first parent of a merge commit was handled
differently and correct but the non-first parents are handled by a
recursive algorithm. Flags should be copied from the root merge commit
to parent-2, to grandparent-2, ... up to the limit of STACK_DEPTH==500
parents-levels. But the recursive algorithm was always copying only to
the direct parents of the merge commit and not the grand*-parents.

This seems to be no problem when commits are handled in a strict date
order because then copying only one level is no problem if children are
handled before parents. But when commits are not seperated anymore by
distinctive correct dates (e.g. because all commits have the same date)
then it may happen that a merge-parent is handled before the merge
commit and when dealing later with the merge commit one has to copy
flags down to more than one level

Bug: 501211
Change-Id: I2d79a7cf1e3bce21a490905ccd9d5e502d7b8421
2016-09-23 11:15:08 +02:00
David Pursehouse d47320bd36 pgm, test: Add missing dependency for buck build
Change-Id: I26d69fb6d35c6fb120360ef143d1b1f565d4014c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-22 15:18:47 +02:00
Matthias Sohn cb617e7ffc Prepare 4.5.1-SNAPSHOT builds
Change-Id: I3305e8a09a3fb06f25a316cff2bdbb551d3ade68
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-21 17:05:27 +02:00
Matthias Sohn e7142a3937 JGit v4.5.0.201609210915-r
Change-Id: Idc02a1a1d74f84605d764c239803f0cfbad94eb7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-21 15:06:18 +02:00
Matthias Sohn ab79ce91c1 Change JGit minimum execution environment to JavaSE-1.8
Bug: 500059
Change-Id: I47f3f6749a67da52029f84e002d9b155ed56d2b7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-20 11:32:36 +02:00
Christian Halstrick bcb5a431a5 Add built-in LFS smudge filter for local case
Adds a JGit built-in implementation of the "git lfs smudge" filter. This
filter should do the same as the one described in [1] besides that it
only supports the local case when the lfs objects are already present in
the media directory. Remote cases where download of LFS objects from an
LFS server is needed will be done in a later commit.

[1] https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-smudge.1.ronn

Change-Id: I8ff661d4edd3667ef7f86f3b4fa33e568eb4c8f4
2016-09-20 10:58:05 +02:00
Christian Halstrick 45ee55d0d9 Add built-in LFS clean filter
Adds a JGit built-in implementation of the "git lfs clean" filter. This
filter should do the same as the one described in [1]. But since this
filter is written in Java and can be called by JGit without forking new
processes it should be much faster

[1]
https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-clean.1.ronn

Change-Id: If60e387e97870245b4bd765eda6717eb84cffb1d
2016-09-20 10:11:27 +02:00
Christian Halstrick d97248467a Add support for built-in smudge filters
JGit supports smudge filters defined in repository configuration. The
filters are implemented as external programs filtering content by
accepting the original content (as seen in git's object database) on
stdin and which emit the filtered content on stdout. This content is
then written to the file in the working tree. To run such a filter JGit
has to start an external process and pump data into/from this process.

This commit adds support for built-in smudge filters which are
implemented in Java and which are executed by jgit's main thread. When a
filter is defined in the configuration as
"jgit://builtin/<filterDriverName>/smudge" then JGit will lookup in a
static map whether a builtin filter is registered under this name. If
found such a filter is called to do the filtering.

The functionality in this commit requires that a program using JGit
explicitly calls the JGit API to register built-in implementations for
specific smudge filters. In follow-up commits configuration parameters
will be added which trigger such registrations.

Change-Id: Ia743aa0dbed795e71e5792f35ae55660e0eb3c24
2016-09-20 10:02:20 +02:00
Christian Halstrick b5bec73715 Add support for built-in clean filters
JGit supports clean filters defined in repository configuration. The
filters are implemented as external programs filtering content by
accepting the original content (as seen in the working tree) on stdin
and which emit the filtered content on stdout. To run such a filter JGit
has to start an external process and pump data into/from this process.

This commit adds support for clean filters which are implemented
in Java and which are executed by jgit's main thread. When a filter is
defined in the configuration as
"jgit://builtin/<filterDriverName>/clean" then JGit will lookup in a
static map whether a filter is registered under this name. If found
such a filter is called to do the filtering.

The functionality in this commit requires that a program using JGit
explicitly calls the JGit API to register built-in implementations for
specific clean filters. In follow-up commits configuration parameters
will be added which trigger such registrations. Other commits will add
implementations for lfs filters.

Change-Id: I0344d3c54801c9a46e5a606c5df17e5f2e17b2be
2016-09-20 09:57:48 +02:00
Matthias Sohn 9d4ffcc1ab Prepare 4.6.0-SNAPSHOT builds
Change-Id: Id2eafc331ee32c332c2a9b867b05c260beb0d10f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-19 17:06:42 +02:00
Thomas Wolf aadbb158e1 Handle all values of branch.[name].rebase
BranchConfig treated this config property as a boolean, but git also
allows the values "preserve" and "interactive". Config property
pull.rebase also allows the same values.

Replace private enum PullCommand.PullRebaseMode by new public enum
BranchConfig.BranchRebaseMode and adapt all uses. Add a new setter to
PullCommand.

Note: PullCommand will treat "interactive" like "true", i.e., as a
non-interactive rebase. Not sure how "interactive" should be handled.
At least it won't balk on it.

Bug: 499482
Change-Id: I7309360f5662b2c2efa1bd8ea6f112c63cf064af
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2016-09-15 00:44:23 +02:00
Andrey Loskutov 619329c84e Ignore trailing spaces in directory rule patterns
Bug: 500967
Change-Id: I7fabc2654af97011c62f46d5c30ee992341e45e2
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2016-09-14 07:43:52 -04:00
Martin Goellnitz 57a263f182 Add support for post-commit hooks
Change-Id: I6691b454404dd4db3c690ecfc7515de765bc2ef7
Signed-off-by: Martin Goellnitz <m.goellnitz@outlook.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-13 17:13:48 +02:00
Matthias Sohn abeaafc9c7 Don't log error if system git config does not exist
- enhance FS.readPipe to throw an exception if the external command
fails to enable the caller to handle the command failure
- reduce log level to warning if system git config does not exist
- improve log message 

Bug: 476639
Change-Id: I94ae3caec22150dde81f1ea8e1e665df55290d42
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-05 17:41:36 +02:00
Matthias Sohn c711361612 Add missing dependency to slf4j-log4j bridge
Without the bridge JGit tests don't show log output in Eclipse console.

Change-Id: I7acce1f1787960b5ca98377cb5c7f599a8a220b5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-05 04:53:14 -04:00
Matthias Sohn a4508f994a Rename FSJava7Test to FSTest
Since 4.0 JGit does no longer support Java versions older than Java 7 so
there is no need anymore to mention Java 7 in the class name.

Change-Id: Ic46c9d89a7e919ae4a69487fa06de0478d2b21f0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-09-03 00:03:00 +02:00
Shawn Pearce c2e2326a43 ReceivePack: refactor push option parsing
Refactor all of the push option support code to allocate the list
immediately before parsing the options section off the stream.

Move option support down to ReceivePack instead of BaseReceivePack.
Push options are specific to the ReceivePack protocol and are not
likely to appear in the 4 year old subscription proposal.  These
changes are OK before JGit 4.5 ships as no consumer should be relying
on these new APIs.

Change-Id: Ib07d18c877628aba07da07cd91875f918d509c49
2016-08-26 14:53:54 -07:00
Stefan Beller 36cf4fe580 Fix push option initalization on HTTP
Initialize pushOptions when we decide to use them, instead of when we
advertise them.

In the case of HTTP the advertisement is in a different network
request, hence in a different instance of the BaseReceivePack.

Change-Id: I094c60942e04de82cb6d8433c9cd43a46ffae332
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-08-26 14:52:07 -07:00
Masaya Suzuki 1227165e27 Clarify the semantics of DfsRefDatabase#compareAndPut
DfsRefDatabase#compareAndPut had a vague semantics for reference
matching. Because of this, an operation to make a symbolic
reference had been broken for some DFS implementations even if they
followed the contract of compareAndPut. The clarified semantics
requires the implementations to satisfy the followings:

* Matching references should be both symbolic references or both
  object ID references.
* If both are symbolic references, both should have the same target
  name.
* If both are object ID references, both should have the same object
  ID.

This semantics is defined based on
https://git.eclipse.org/r/#/c/77416/. Before this commit,
DfsRefDatabase couldn't see the target of symbolic references.

InMemoryRepository is changed to comply with the new semantics. This
semantics change can affect the existing DFS implementations that only
checks object IDs. This commit adds two tests that the previous
InMemoryRepository couldn't pass.

Change-Id: I6c6b5d3cc8241a81f4a37782381c88e8a59fdf15
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2016-08-25 19:12:39 -07:00
Dave Borowitz 649ad06586 NoteMapTest: Add missing @Test annotations
The RepositoryTestCase hierarchy no longer comes from TestCase, so all
test methods must have @Test.

Fix one test that was broken but never run; fortunately this was just
a typo in the test code.

Change-Id: I3ac8ccdab5e2d5539c63d7b0a88d8bdb0c5ff66e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-08-26 00:12:32 +02:00
Masaya Suzuki edd8ad4d04 Use FS#lastModified instead of File#lastModified
This fixes the tests failed in JDK8.

FS uses java.nio API to get file attributes. The timestamps obtained
from that API are more precise than the ones from
java.io.File#lastModified() since Java8.

This difference accidentally makes JGit detect newly added files as
smudged. Use the precised timestamp to avoid this false positive.

Bug: 500058
Change-Id: I9e587583c85cb6efa7562ad6c5f26577869a2e7c
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2016-08-24 09:07:08 +02:00
Andrey Loskutov b8260b5e79 Added Java 7 launch config with LANG env. variable set
This avoids symlink test errors on Linux

Change-Id: Id8193524c40394a90b8315ab0b8256670d618cb5
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2016-08-24 09:05:22 +02:00
Matthias Sohn 13f0db25f2 Enhance ResetCommand to allow disabling reflog update
This will be used by EGit for implementing commit amend in the staging
view (see Idcd1efeeee8b3065bae36e285bfc0af24ab1e88f).

Change-Id: Ice9ebbb1c0c3314c679f4db40cdd3664f61c27c3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-08-17 08:56:03 -04:00
Matthias Sohn 63bfd9b976 Merge "Add path src/ to source path in build.properties" 2016-08-08 02:42:22 -04:00
Christian Halstrick 2c5c035e62 Merge "Skip cleaning inner repositories by default in CleanCommand" 2016-08-07 09:37:48 -04:00
Christian Halstrick cb3bd8ea84 Merge "Add testCleanDirsWithSubmodule test to CleanCommandTest" 2016-08-07 09:37:22 -04:00
Matthias Sohn 8141140922 Add path src/ to source path in build.properties
This fixes the warning "src/ is missing from source.."

Change-Id: I166e3a6a3d5230e4110d3283ec4dbc7d1dfe6732
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-08-06 19:12:06 -04:00
Jonathan Nieder 1c245da5ac Merge changes I27961679,I91be6165,If0dbd562
* changes:
  LfsProtocolServlet: Allow access to objects in request
  LfsProtocolServlet: Allow getLargeFileRepository to raise exceptions
  Remove references to org.eclipse.jgit.java7
2016-08-05 20:36:09 -04:00
Terry Parker e426aa8b90 Shallow fetch/clone: Make --depth mean the total history depth
cgit changed the --depth parameter to mean the total depth of history
rather than the depth of ancestors to be returned [1]. JGit still uses
the latter meaning, so update it to match cgit.

depth=0 still means a non-shallow clone. depth=1 now means only the
wants rather than the wants and their direct parents.

This is accomplished by changing the semantic meaning of "depth" in
UploadPack and PackWriter to mean the total depth of history desired,
while keeping "depth" in DepthWalk.{RevWalk,ObjectWalk} to mean
the depth of traversal. Thus UploadPack and PackWriter always
initialize their DepthWalks with "depth-1".

[1] upload-pack: fix off-by-one depth calculation in shallow clone
https://code.googlesource.com/git/+/682c7d2f1a2d1a5443777237450505738af2ff1a

Change-Id: I87ed3c0f56c37e3491e367a41f5e555c4207ff44
Signed-off-by: Terry Parker <tparker@google.com>
2016-08-05 17:27:30 -07:00
Terry Parker d385a7a5e5 Shallow fetch: Respect "shallow" lines
When fetching from a shallow clone, the client sends "have" lines
to tell the server about objects it already has and "shallow" lines
to tell where its local history terminates. In some circumstances,
the server fails to honor the shallow lines and fails to return
objects that the client needs.

UploadPack passes the "have" lines to PackWriter so PackWriter can
omit them from the generated pack. UploadPack processes "shallow"
lines by calling RevWalk.assumeShallow() with the set of shallow
commits. RevWalk creates and caches RevCommits for these shallow
commits, clearing out their parents. That way, walks correctly
terminate at the shallow commits instead of assuming the client has
history going back behind them. UploadPack converts its RevWalk to an
ObjectWalk, maintaining the cached RevCommits, and passes it to
PackWriter.

Unfortunately, to support shallow fetches the PackWriter does the
following:

  if (shallowPack && !(walk instanceof DepthWalk.ObjectWalk))
    walk = new DepthWalk.ObjectWalk(reader, depth);

That is, when the client sends a "deepen" line (fetch --depth=<n>)
and the caller has not passed in a DepthWalk.ObjectWalk, PackWriter
throws away the RevWalk that was passed in and makes a new one. The
cleared parent lists prepared by RevWalk.assumeShallow() are lost.
Fortunately UploadPack intends to pass in a DepthWalk.ObjectWalk.
It tries to create it by calling toObjectWalkWithSameObjects() on
a DepthWalk.RevWalk. But it doesn't work: because DepthWalk.RevWalk
does not override the standard RevWalk#toObjectWalkWithSameObjects
implementation, the result is a plain ObjectWalk instead of an
instance of DepthWalk.ObjectWalk.

The result is that the "shallow" information is thrown away and
objects reachable from the shallow commits can be omitted from the
pack sent when fetching with --depth from a shallow clone.

Multiple factors collude to limit the circumstances under which this
bug can be observed:

1. Commits with depth != 0 don't enter DepthGenerator's pending queue.
   That means a "have" cannot have any effect on DepthGenerator unless
   it is also a "want".

2. DepthGenerator#next() doesn't call carryFlagsImpl(), so the
   uninteresting flag is not propagated to ancestors there even if a
   "have" is also a "want".

3. JGit treats a depth of 1 as "1 past the wants".

Because of (2), the only place the UNINTERESTING flag can leak to a
shallow commit's parents is in the carryFlags() call from
markUninteresting(). carryFlags() only traverses commits that have
already been parsed: commits yet to be parsed are supposed to inherit
correct flags from their parent in PendingGenerator#next (which
doesn't happen here --- that is (2)). So the list of commits that have
already been parsed becomes relevant.

When we hit the markUninteresting() call, all "want"s, "have"s, and
commits to be unshallowed have been parsed. carryFlags() only
affects the parsed commits. If the "want" is a direct parent of a
"have", then it carryFlags() marks it as uninteresting. If the "have"
was also a "shallow", then its parent pointer should have been null
and the "want" shouldn't have been marked, so we see the bug. If the
"want" is a more distant ancestor then (2) keeps the uninteresting
state from propagating to the "want" and we don't see the bug. If the
"shallow" is not also a "have" then the shallow commit isn't parsed
so (2) keeps the uninteresting state from propagating to the "want
so we don't see the bug.

Here is a reproduction case (time flowing left to right, arrows
pointing to parents). "C" must be a commit that the client
reports as a "have" during negotiation. That can only happen if the
server reports it as an existing branch or tag in the first round of
negotiation:

  A <-- B <-- C <-- D

First do

  git clone --depth 1 <repo>

which yields D as a "have" and C as a "shallow" commit. Then try

  git fetch --depth 1 <repo> B:refs/heads/B

Negotiation sets up: have D, shallow C, have C, want B.
But due to this bug B is marked as uninteresting and is not sent.

Change-Id: I6e14b57b2f85e52d28cdcf356df647870f475440
Signed-off-by: Terry Parker <tparker@google.com>
2016-08-05 18:37:36 -04:00
David Pursehouse 1ce24d46a6 Remove references to org.eclipse.jgit.java7
The bundle org.eclipse.jgit.java7 was removed in 4.0.

Remove references to it from the README.md.

Remove reference to it from org.eclipse.jgit.test/.project, which
causes an error message when opening the project in Eclipse:

  Resource '/org.eclipse.jgit.java7' does not exist.

Change-Id: If0dbd562dcd60550bec3c0f793289474b7624bce
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-08-05 11:22:27 +09:00
Terry Parker 7edf05530d Shallow fetch: avoid sending unneeded blobs
When doing an incremental fetch from JGit, "have" commits are marked
as "uninteresting". In a non-shallow fetch, when the RevWalk hits an
"uninteresting" commit it marks the commit's corresponding tree as
uninteresting. That has the effect of dropping those trees and all the
trees and blobs they reference out of the thin pack returned to the
client.

However, shallow fetches use a DepthWalk to limit the RevWalk, which
nearly always causes the RevWalk to terminate before encountering the
"have" commits. As a result the pack created for the incremental fetch
never encounters "uninteresting" tree objects and thus includes
duplicate objects that it knows the client already has.

Change-Id: I7b1f7c3b0d83e04d34cd2fa676f1ad4fec904c05
Signed-off-by: Terry Parker <tparker@google.com>
2016-08-04 17:20:23 -07:00
Terry Parker 9843489cb8 PackWriterTest: Improve readability
Add wants() and haves() static utility functions to improve readability.


Change-Id: I4d44e17a9af97c0203e2ebe112eabb1f67d272a6
Signed-off-by: Terry Parker <tparker@google.com>
2016-08-04 14:59:05 -07:00
Matthaus Owens e1ffab1cac Skip cleaning inner repositories by default in CleanCommand
Previously jgit would attempt to clean git repositories that had not
been committed by calling a non-recursive delete on them, which would
fail as they are directories. This commit addresses that issue in the
following ways.
Repositories are skipped in a default clean, similarly to cgit and only
cleaned when the force flag is applied. When the force flag is applied
repositories are deleted using a recursive delete call. The force flag
and setForce method are added here to CleanCommand to support this
change.

Bug: 498367
Change-Id: Ib6cfff65a033d0d0f76395060bf76719e13fc467
Signed-off-by: Matthaus Owens <matthaus@puppetlabs.com>
2016-08-04 13:38:52 -07:00
Matthaus Owens 0ee89f01be Add testCleanDirsWithSubmodule test to CleanCommandTest
This commit adds some test coverage to cleaning a repository with a
submodule, which did not previously exist.

Bug: 498367
Change-Id: Ia5c4e4cc53488800dd486f8556dc57656783f1c4
Signed-off-by: Matthaus Owens <matthaus@puppetlabs.com>
2016-08-04 13:04:17 +01:00
David Pursehouse 0d872a0837 Annotate Sets#of with @SafeVarArgs to prevent heap pollution warning
This prevents the warning:

  Potential heap pollution via varargs parameter

The method doesn't do any casting of types that would cause the heap
pollution, so it should be safe to add @SafeVarArgs.

See [1] for information about this warning.

[1] http://stackoverflow.com/a/12462259/381622

Change-Id: Ic6d252915ea44b4f1c385afecb98906cd2c54382
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 18:58:17 +09:00
David Pursehouse 4528d5f56d Ignore 'The value of exception parameter is not used' warning
Change-Id: I50407e4a33e35b718ca40503fdd436f1f9f70fba
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 10:16:49 +09:00
David Pursehouse 0be8021819 Merge branch 'stable-4.4'
* stable-4.4:
  JGit v4.4.1.201607150455-r
  RefDirectory: remove ref lock file for following ref dir removal

Change-Id: Ifc8a782efd7f2f991e70ad2a3691a8dba66c7554
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 10:09:35 +09:00
Stefan Beller a2d3c376a6 RefSpecs: allow construction of weird wildcarded RefSpecs
Gerrit's superproject subscription feature uses RefSpecs to formalize
the ACLs of when the superproject subscription feature is allowed.

As this is a slightly different use case than describing a local/remote
pair of refs, we need to be more permissive. Specifically we want to allow:

    refs/heads/*
    refs/heads/*:refs/heads/master
    refs/heads/master:refs/heads/*

Introduce a new constructor, that allows constructing these RefSpecs.

Change-Id: I46c0bea9d876e61eb2c8d50f404b905792bc72b3
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-07-25 10:10:06 -07:00
Stefan Beller 1556ca740b RefSpec: reject refs ending in '/'
We had a case in Gerrits superproject subscriptions where
'refs/heads/' was configured with the intention to mean 'refs/heads/*'.

The first expression lacks the '*', which is why it is not considered
a wildcard but it was considered valid and so was not found early to be
a typo.

Refs are not allowed to end with '/' anyway, so add a check for that.

Change-Id: I3ffdd9002146382acafb4fbc310a64af4cc1b7a9
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-07-25 10:10:06 -07:00
Dan Wang 7f9fb80002 Push implementation of option strings
Example usage:
$ ./jgit push \
  --push-option "Reviewer=j.doe@example.org" \
  --push-option "<arbitrary string>" \
  origin HEAD:refs/for/master
Stefan Beller has also made an equivalent change to CGit:
http://thread.gmane.org/gmane.comp.version-control.git/299872

Change-Id: I6797e50681054dce3bd179e80b731aef5e200d77
Signed-off-by: Dan Wang <dwwang@google.com>
2016-07-22 15:40:54 -07:00
Matthias Sohn df479df176 JGit v4.4.1.201607150455-r
Change-Id: I61dbc29a962c8185fb356fe1ca30a1e673166d47
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-15 10:54:36 +02:00
Christian Halstrick eaedbc3980 Merge "RefDirectory: remove ref lock file for following ref dir removal" into stable-4.4 2016-07-15 04:43:53 -04:00
Matthias Sohn ffddf8d437 Merge branch 'stable-4.4'
* stable-4.4:
  Log if Repository.useCnt becomes negative
  Time based eviction strategy for repository cache
  Add method to read time unit from config
  Align include.path max depth with native git
  Config load should not fail on unsupported or nonexistent include path
  Allow using JDK 7 bootclasspath when compiling JGit using Java 8
  Extract work queue to allow reusing it

Change-Id: I6aeedb1cb8b0c3068af344a719c80a03ae68fc23
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-12 17:12:41 +02:00
Christian Halstrick 7ffe547da7 Time based eviction strategy for repository cache
When Repository.close() decrements the useCount to 0 currently the cache
immediately evicts the repository from WindowCache and RepositoryCache.
This leads to I/O overhead on busy repositories because pack files and
references are inserted and deleted from the cache frequently.

This commit defers the eviction of a repository from the caches until
last use of the repository is older than time to live. The eviction is
handled by a background task running periodically.

Add two new configuration parameters:
* core.repositoryCacheExpireAfter: cache entries are evicted if the
cache entry wasn't accessed longer than this time in milliseconds
* core.repositoryCacheCleanupDelay: defines the interval in milliseconds
for running a background task evicting expired cache entries. If set to
-1 the delay is set to min(repositoryCacheExpireAfter, 10 minutes). If
set to 0 the time based eviction is switched off and no background task
is started. If time based eviction is switched off the JVM can still
evict cache entries if heap memory is running low.

Change-Id: I4a0214ad8b4a193985dda6a0ade63b70bdb948d7
Also-by: Matthias Sohn <matthias.sohn@sap.com>
Also-by: Hugo Arès <hugo.ares@ericsson.com>
Also-by: Sasa Zivkov <sasa.zivkov@sap.com>
2016-07-12 11:32:49 +02:00
Hugo Arès c1ca9cc800 Add method to read time unit from config
Time units supported:

-milliseconds (1 ms, 2 milliseconds)
-seconds (1 s, 1 sec, 1 second, 2 seconds)
-minutes (1 m, 1 min, 1 minute, 2 minutes)
-hours (1 h, 1 hr, 1 hour, 2 hours)
-days (1 d, 1 day, 2 days)
-weeks (1 w, 1 week, 2 weeks)
-months (1 mon, 1 month, 2 months)
-years (1 y, 1 year, 2 years)

This functionality is implemented in Gerrit ConfigUtil class. Add it to
JGit so it can eventually be remove from Gerrit.

Change-Id: I2d6564ff656b6ab9424a9360624061c94fd5f413
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-12 11:32:49 +02:00
Matthias Sohn 450c0af42f Fix unclosed resource warnings in FileTreeIteratorTest
Change-Id: I75ea7deca64a707cd6b5c61c3c83062f20041684
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-07 17:36:55 -04:00
Shawn Pearce 63cf87f863 ReceivePack: report protocol parsing failures on channel 3
If the client sent a well-formed enough request to see it wants to use
side-band-64k for status reporting (meaning its a modern client), but
any other command record was somehow invalid (e.g. corrupt SHA-1)
report the parsing exception using channel 3.  This allows clients to
see the failure and know the server will not be continuing.

git-core and JGit clients send all commands and then start a sideband
demux before sending the pack. By consuming all commands first we get
the client into a state where it can see and respond to the channel 3
server failure.

This behavior is useful on HTTPS connections when the client is buggy
and sent a corrupt command, but still managed to request side-band-64k
in the first line.

Change-Id: If385b91ceb9f024ccae2d1645caf15bc6b206130
2016-07-05 17:22:37 -07:00
Shawn Pearce ae5870a1f2 PushConnectionTest: Use Java 7 try-with-resources
Change-Id: I429066d2af9c5fb8bf8bc038b99ebe5a7dc09170
2016-07-05 12:32:24 -07:00
Christian Halstrick d7a8831785 Enhance test coverage when core.filemode is false
Some branches in WorkingTreeIterator.getIndexFileMode() have not been
covered by tests. Enhance the tests to increase test coverage.

Change-Id: I400a221048f0f6cbaa987350eaf998b0ebb50a4e
2016-07-01 22:48:29 +02:00
Shawn Pearce ec5190e2e3 Merge changes from topic 'dfs-gc'
* changes:
  Prune UNREACHABLE_GARBAGE packs when they expire
  Use try-with-resources in DfsGarbageCollector.writePack
  Fix lastModified to be consistent in DfsGarbageCollector
  Add GC_REST PackSource to better order DFS packs
2016-06-27 11:52:01 -04:00
Mike Williams fd527a2cd7 Prune UNREACHABLE_GARBAGE packs when they expire
DfsGarbageCollector will now enforce a maximum time to live (TTL) for
UNREACHABLE_GARBAGE packs.  The default TTL is 1 day, which should be
enough time to avoid races with other processes that are inserting
data into the repository.

Change-Id: Id719e6e2a03cfc9a0c0aef8ed71d261dda14bd0c
Signed-off-by: Mike Williams <miwilliams@google.com>
2016-06-27 08:38:11 -07:00
Hugo Arès 80cd855443 Config load should not fail on unsupported or nonexistent include path
1f86350 added initial support for include.path. Relative path and path
with tilde are not yet supported but config load was failing if one of
those 2 unsupported options was encountered. Another problem was that
config load was failing if the include.path file did not exist.

Change the behavior to be consistent with native git. Ignore unsupported
or nonexistent include.path.

Bug: 495505
Bug: 496732
Change-Id: I7285d0e7abb6389ba6983e9c46021bea4344af68
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
2016-06-27 08:59:03 -04:00
Christian Halstrick e7df61ef4d Fix TreeWalk to reset attributes cache for each entry
Treewalk has a member 'attr' which caches the attributes for the current
entry. We did not reset the cache always when moving to next entry. The
effect was that when there are no attributes for an entry 'a' but 'a'
was skipped by a Treewalk filter then Treewalk stopped looking for
attributes until TreeWalk.next() was called again.

Change-Id: Ied39b7fb5f56afe7a237da17801003d0abe6b1c7
2016-06-23 17:11:47 +02:00
Christian Halstrick 5fe44ed3ee Fix DirCacheCheckout to return CheckoutConflictException
Problem occurs when the checkout wants to create a file 'd/f' but
the workingtree contains a dirty file 'd'. In order to create d/f the
file 'd' would have to be deleted and since the file is dirty that
content would be lost. This should lead to a CheckoutConflictException
for d/f when failOnConflict was set to true.

This fix also changes jgit checkout semantics to be more like native
gits checkout semantics. If during a checkout jgit wants to delete a
folder but finds that the working tree contains a dirty file at this
path then JGit will now throw an exception instead of silently keeping
the dirty file. Like in this example:

	git init
	touch b
	git add b
	git commit -m addB
	mkdir a
	touch a/c
	git add a/c
	git commit -m addAC
	rm -fr a
	touch a
	git checkout HEAD~

Change-Id: I9089123179e09dd565285d50b0caa308d290cccd
Signed-off-by: Rüdiger Herrmann <ruediger.herrmann@gmx.de>
Also-by: Rüdiger Herrmann <ruediger.herrmann@gmx.de>
2016-06-23 09:34:22 +02:00
Shawn Pearce 2ec3accb3b Optimize RefAdvertiser for wire protocol
The native wire protocol sends ref advertisements in the pkt-line
format, which requires encoding the ObjectId and ref name onto a byte
sequence.  Busy servers show this is a very high source of garbage,
which pushes the garbage collector harder when there are many refs in
the repository (e.g.  70k, in a Gerrit managed repository).

Optimize the side band advertiser by retaining the CharsetEncoder,
minimizing the amount of temporary garbage built during encoding.

Change-Id: I406c654bf82c1eb94b38862da2425e98396134cb
2016-06-18 21:41:07 -07:00
Matthias Sohn 48cb5ffba6 Merge branch 'stable-4.4'
* stable-4.4:
  Prepare 4.4.1-SNAPSHOT builds
  JGit v4.4.0.201606070830-r
  Prepare 4.4.0-SNAPSHOT builds
  JGit v4.4.0.201606011500-rc2
  Prepare 4.4.0-SNAPSHOT builds
  JGit v4.4.0.201605250940-rc1
  Update Orbit repository for Neon to R20160520211859
  Fix computation of id in WorkingTreeIterator with autocrlf and 
    smudging
  Prepare 4.3.2-SNAPSHOT builds
  JGit v4.3.1.201605051710-r
  Prepare 4.4.0-SNAPSHOT builds
  JGit v4.4.0.201605041135-m1
  Run Maven build in release.sh concurrently to speedup release

Change-Id: I25ef0497a4455b8229b453e1023abb4631d4b6d3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-14 00:11:37 +02:00
Marco Miller 734d5bb3c9 RefDirectory: remove ref lock file for following ref dir removal
Before this fix, ref directory removal did not work. That was because
the ref lock file was still in the leaf directory at deletion time.
Hence no deep ref directories were ever deleted, which negatively
impacted performance under large directory structure circumstances.

This fix removes the ref lock file before attempting to delete the ref
directory (which includes it). The other deep parent directories are
therefore now successfully deleted in turn, since leaf's content
(lock file) gets removed first.

So, given a structure such as refs/any/directory[/**], this fix now
deletes all empty directories up to -and including- 'directory'. The
'any' directory (e.g.) does not get deleted even if empty, as before.

The ref lock file is still also removed in the calling block's finally
clause, just in case, as before. Such double-unlock brought by this
fix is harmless (a no-op).

A new (private) RefDirectory#delete method is introduced to support
this #pack-specific case; other RefDirectory#delete callers remain
untouched.

Change-Id: I47ba1eeb9bcf0cb93d2ed105d84fea2dac756a5a
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
2016-06-10 08:49:53 -04:00
Shawn Pearce 7e8637658c Merge "Add method to read time unit from config" 2016-06-10 00:26:46 -04:00
Matthias Sohn 0900dc08f5 Prepare 4.4.1-SNAPSHOT builds
Change-Id: I4fe1c84b1418126f5b245b8a4f72bb7e37f48eae
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-07 17:30:33 +02:00
Matthias Sohn 399d2929ee JGit v4.4.0.201606070830-r
Change-Id: I5c771a62ac0420424a655a4025c62a4bc28dc44e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-07 13:51:02 +02:00
Christian Pontesegger ac3d3af632 http transport does not use authentication fallback
Git servers supporting HTTP transport can send multiple WWW-Authenticate
challenges [1] for different authentication schemes the server supports.
If authentication fails now retry all authentication types proposed by
the server.

[1] https://tools.ietf.org/html/rfc2617#page-3

Bug: 492057
Change-Id: I01d438a5896f9b1008bd6b751ad9c7cbf780af1a
Signed-off-by: Christian Pontesegger <christian.pontesegger@web.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-06 16:45:49 +02:00
Hugo Arès f379688aad Add method to read time unit from config
Time units supported:

-milliseconds (1 ms, 2 milliseconds)
-seconds (1 s, 1 sec, 1 second, 2 seconds)
-minutes (1 m, 1 min, 1 minute, 2 minutes)
-hours (1 h, 1 hr, 1 hour, 2 hours)
-days (1 d, 1 day, 2 days)
-weeks (1 w, 1 week, 2 weeks)
-months (1 mon, 1 month, 2 months)
-years (1 y, 1 year, 2 years)

This functionality is implemented in Gerrit ConfigUtil class. Add it to
JGit so it can eventually be remove from Gerrit.

Change-Id: I2d6564ff656b6ab9424a9360624061c94fd5f413
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
2016-06-06 10:08:37 -04:00
Christian Halstrick efc1f7a112 Merge "Add missing @Test annotation in RepositoryCacheTest" 2016-06-02 03:30:03 -04:00
Christian Halstrick df75b9c0c1 Merge "Add missing @Test annotations" 2016-06-02 03:28:02 -04:00
Matthias Sohn 34626614a8 Merge "Prepare 4.5.0-SNAPSHOT builds" 2016-06-01 18:13:35 -04:00
Dave Borowitz 0d6ba84065 DfsInserter: Optionally disable existing object check
When using a DfsInserter for high-throughput insertion of many
objects (analogous to git-fast-import), we don't necessarily want to
do a random object lookup for each. It'll be faster from the
inserter's perspective to insert the duplicate objects and let a later
GC handle the deduplication.

Change-Id: Ic97f5f01657b4525f157e6df66023f1f07fc1851
2016-06-01 17:21:33 -04:00
Matthias Sohn 75b3301939 Prepare 4.5.0-SNAPSHOT builds
Change-Id: I572fe9fea0e5ca0bec4648c916ae95a5b1ccf125
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-01 23:04:07 +02:00
Matthias Sohn 831c811f61 Prepare 4.4.0-SNAPSHOT builds
Change-Id: Ib41dfe4103f013ba87a1ce62ee24bc9e10300eff
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-01 22:36:51 +02:00
Hugo Arès f0a485d5a2 Add missing @Test annotations
Change-Id: Ie5ffcbf18233dc9b3eb232f714b5b58b6b93f4e0
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
2016-06-01 16:06:23 -04:00
Hugo Arès 1c1885b81d Add missing @Test annotation in RepositoryCacheTest
Change-Id: I25a9b294084f7b9769efa67001a4973647da4079
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
2016-06-01 15:59:07 -04:00
Matthias Sohn f3b8653021 JGit v4.4.0.201606011500-rc2
Change-Id: Ifd7b7baba7d676a630e2f742d369c4186ea450a6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-01 20:37:53 +02:00
Matthias Sohn dfb33e699c Merge branch 'master' into stable-4.4
* master:
  Fix javadoc errors and unused imports introduced by ddd0fe25
  RepoCommand: record manifest shallow recommendation in .gitmodules
  RepoCommand: record manifest groups as submodule labels
  Remove the deprecated TestRepository.getClock() method
  Replace use of deprecated method Repository.getRef()
  [findBugs] Prevent potential NPE in
    FileLfsRepository.getOutputStream()
  Better report on client side if push failed due to too large object
  [findBugs] Prevent potential NPE in CloneCommand.init()
  RepoCommand: remove --record-remote-branches
  RepoCommandTest: Improve assertion message for remote branch recording

Change-Id: I4fbce4f84925a933fcc9a48058ed6793f5821b97
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-01 16:10:11 +02:00
Matthias Sohn 525baa1213 Fix javadoc errors and unused imports introduced by ddd0fe25
Change-Id: I2d17115843e453c71febd4a58527effe658b48fd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-06-01 10:17:29 +02:00
Stefan Beller 6ae4ed0f7d RepoCommand: record manifest shallow recommendation in .gitmodules
Git core learned about the submodule.<name>.shallow option in
.gitmodules files, which is a recommendation to clone a submodule
shallow. A repo manifest may record a clone depth recommendation as
an optional field, which contains more information than a binary
shallow/nonshallow recommendation, so any attempted conversion may be
lossy. In practice the clone depth recommendation is either '1' or doesn't
exist, which is the binary behavior we have in Git core.

Change-Id: I51aa9cb6d1d9660dae6ab6d21ad7bae9bc5325e6
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-05-31 15:19:52 -07:00
Stefan Beller ddd0fe257b RepoCommand: record manifest groups as submodule labels
Git core learned about attributes in pathspecs:

    pathspec: allow querying for attributes

    The pathspec mechanism is extended via the new
    ":(attr:eol=input)pattern/to/match" syntax to filter paths so that it
    requires paths to not just match the given pattern but also have the
    specified attrs attached for them to be chosen.

(177161a5f7, 2016-05-20)

We intend to use these pathspec attribute patterns for submodule
grouping, similar to the grouping in repo. So the RepoCommand which
translates repo manifest files into submodules should propagate this
information along. This requires writing information to the
.gitattributes file instead of the .gitmodules file. For now we just
overwrite any existing .gitattributes file and do not care about prior
attributes set. If this becomes an issue we need to figure out how to
correctly amend the grouping information to an existing .gitattributes
file.

Change-Id: I0f55b45786b6b8fc3d5be62d7f6aab9ac00ed60e
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-05-31 15:18:20 -07:00
Matthias Sohn 4c236ff4bb Replace use of deprecated method Repository.getRef()
Change-Id: Iecf2b8deafc4991cc3333702fb9fa0638be7b914
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-31 00:11:11 +02:00
Stefan Beller 53a64a0b8c RepoCommandTest: Improve assertion message for remote branch recording
Change-Id: Ic14e9f9c4bf007b464e463f5047ec5a8d0b9236c
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-05-25 16:57:42 -07:00
Matthias Sohn 9720f9e249 Prepare 4.4.0-SNAPSHOT builds
Change-Id: Iba6e423294d3315465648d19087a674a73b5ef28
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-25 23:30:55 +02:00
Matthias Sohn f2c58503d7 JGit v4.4.0.201605250940-rc1
Change-Id: Idd8e1664730b979cfbd9a2196081a48a1efa3330
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-25 15:14:46 +02:00
Matthias Sohn 88524581cf Merge branch 'master' into stable-4.4
* master:
  JGit CLI: allow to call git init with specific directory
  Redirect all Show output to outs
  Support git config [include] section with absolute path(s)
  Added filter for merge and non-merges commits.
  [findBugs] Prevent potential NPE in FS_POSIX.readUmask()
  [findBugs] Fix calculation of host header in SignerV4
  Update Orbit repository to S20160518051658 for Neon RC2
  Fix StashApply regarding handling of untracked files
  GC should not pack objects only referenced by ORIG_HEAD,...
  Make sure to overwrite files when "reset --hard" detects conflicts
  Allow setting FileMode to executable when applying patches in 
    ApplyCommand
  Fix config value get to return last instead of 1st just like git
  Remove UTF-8 checking duplication in Config lib subclasses
  Update Maven plugins
  Fix type parameter in javadoc in TestRepository.delete(String ref)
  TestRepository: Add delete() method
  Make BaseReceivePack.setAtomic public
  ReceivePack: Pass atomic setting from client to BatchRefUpdate

Change-Id: I5c9c5b7ccb23fb48b44b3da10b2c5d876d043d24
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-24 17:00:08 +02:00
Matthias Sohn fc0ec94bc3 Merge branch 'stable-4.3' into stable-4.4
* stable-4.3:
  Fix computation of id in WorkingTreeIterator with autocrlf and
    smudging
  Prepare 4.3.2-SNAPSHOT builds
  JGit v4.3.1.201605051710-r
  Scan loose ref before packed in case gc about to remove the loose
  Fix possible NPEs when reporting transport errors
  Fix calling of clean/smudge filters from Checkout,MergeCommands
  Fix ApplyCommand when result of patch is an empty file

Change-Id: I829f06699f6670e519d04c927bdba4b82df29199
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-24 16:44:03 +02:00
Christian Halstrick ffa237e7db Fix computation of id in WorkingTreeIterator with autocrlf and smudging
JGit failed to do checkouts when the index contained smudged entries and
autocrlf was on. In such cases the WorkingTreeIterator calculated the
SHA1 sometimes on content which was not correctly filtered. The SHA1 was
computed on content which two times went through a lf->crlf conversion.

We used to tell the treewalk whether it is a checkin or checkout
operation and always use the related filters when reading any content.
If on windows and autocrlf is true and we do a checkout operation then
we always used a lf->crlf conversion on any text content. That's not
correct. Even during a checkout we sometimes need the crlf->lf
conversion. E.g. when calculating the content-id for working-tree
content we need to use crlf->lf filtering although the overall operation
type is checkout.

Often this bug does not have effects because we seldom compute the
content-id of filesystem content during a checkout. But we do need to
know whether a file is dirty or not before we overwrite it during a
checkout. And if the index entries are smudged we don't trust the index
and compute filesystem-content-sha1's explicitly.

This caused EGit not to be able to switch branches anymore on Windows
when autocrlf was true. EGit denied the checkout because it thought
workingtree files are dirty because content-sha1 are computed on wrongly
filtered content.

Bug: 493360
Change-Id: I1072a57b4c529ba3aaa50b7b02d2b816bb64a9b8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-23 15:39:04 +02:00
Marco Miller 1f86350c5a Support git config [include] section with absolute path(s)
As per [1], but limited to absolute paths indeed. No support yet for
tilde or $HOME expansion. Support for the --[no-]includes options
([1]) is not part of this commit scope either, but those options'
defaults are in effect as described in [1].

[1] https://git-scm.com/docs/git-config

Included path can be a config file that includes other path-s in turn.
An exception is thrown if too many recursions (circular includes)
happen because of ill-specified config files.

Change-Id: I700bd7b7e1625eb7de0180f220c707d8e7b0930b
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-21 01:14:06 +02:00
Alcemir Santos ba0dfe1ae2 Added filter for merge and non-merges commits.
Added the option to retrieve either merge or non-merge commits in the
LogCommand.

Change-Id: Ie0e1c515a823f2392783f1a47d385c31230e8167
Signed-off-by: Alcemir Santos <alcemir.santos@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-20 01:05:40 +02:00
Matthias Sohn 4641c43a98 Prepare 4.3.2-SNAPSHOT builds
Change-Id: I71997a8be83841765111cb7b55506bd8edd8adad
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-19 10:17:39 +02:00
Christian Halstrick ad1548b49e Fix StashApply regarding handling of untracked files
There was a bug regarding how JGit handled untracked files when applying
a stash. Problem was that untracked files are applied by doing a merge
of HEAD and untrackedFiles commit with a merge base of the stashed HEAD.
That's wrong because the untrackedFiles commit has no parent and
contains only the untracked files. Using stashed HEAD as merge base
leads to unneccessary conflicts on files not event included in the
untrackedFiles commit.

Imagine this graph directly before you want to apply a stash which was
based on 0. You want to apply the stash on current HEAD commit 5.

  5 (HEAD,master)
 /
0---+
 \   \
  1---3 (WIP on master)
     /
    2 (untracked files on master)

Imagine for a specific (tracked) file f
- commit 0 contains X
- HEAD contains Y
- commit 2 (the untracked files) does not contain file f

A merge of 2 and 5 with a merge base of 0 leads to a conflict. The 5
commit wants to modify the file and the 2 commit wants to delete the
file -> conflict.

If no merge base is set then the semantic is correct.

Thanks to Bow for finding this bug and providing the test case.

Bug: 485467
Change-Id: I453fa6ec337f81b2a52c4f51f23044faeec409e6
Also-by: Bow Ruggeri <bow@bow.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-19 00:04:32 +02:00
Christian Halstrick 0afd62efa8 Make sure to overwrite files when "reset --hard" detects conflicts
When performing a "reset --hard" a checkout is done. The pathes are
checked for potential checkout conflicts. But in the end for all
remaining conflicts these files are simply deleted from the working
tree. That's not the right strategy to handle checkout conflicts during
"reset --hard". Instead for every conflict we should simply checkout the
merge commit's content.

This is different from native gits behavior which reports errors when
during a "checkout --hard" a file shows up where a folder was expected.

  "warning: unable to unlink d/c.txt: Not a directory"

Why it is like that in native git was asked in
http://permalink.gmane.org/gmane.comp.version-control.git/279482. Unless
it is explained why native git why this error is reported JGit should
overwrite the files.

Bug: 474842
Change-Id: I08e23822a577aaf22120c5137eb169b6bd08447b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-18 13:29:02 +02:00
Nadav Cohen e81592e076 Allow setting FileMode to executable when applying patches in ApplyCommand
git-apply allows modifying file modes in patched files using either
"new mode" or "new file mode" headers. This patch adds support for
setting files as executables and vice-versa.

Change-Id: I24848966b46f686f540a8efa8150b42e0d9c3ad1
Signed-off-by: Nadav Cohen <nadavcoh@gmail.com>
2016-05-17 00:08:01 -07:00
Marco Miller 00db4ab06e Fix config value get to return last instead of 1st just like git
Before this fix, getting the value of 'key' below used to return
value1. This fix makes it so that value3 gets returned instead,
just like native git's get.

[section]
  key = value1
  key = value2
  key = value3

Change-Id: Iccb24de9b63c3ad8646c909494ca3f8c9ed6e29c
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-12 00:26:38 +02:00
Matthias Sohn 41b194c718 JGit v4.3.1.201605051710-r
Change-Id: I25fb72cc17d7a27b0b69746d907dac9ccd4d7997
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-05 23:08:18 +02:00
Matthias Sohn 89db7e01fa Merge branch 'stable-4.3'
* stable-4.3:
  Scan loose ref before packed in case gc about to remove the loose
  Fix possible NPEs when reporting transport errors
  Fix calling of clean/smudge filters from Checkout,MergeCommands
  Fix ApplyCommand when result of patch is an empty file

Change-Id: I0dc76b7a8c87ce8e0386a1b9e0fa92a3aa62abf7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-05 01:13:45 +02:00
Matthias Sohn b4c6a41adc Prepare 4.4.0-SNAPSHOT builds
Change-Id: Ib95be1300b1b95cc7e8153170d509282bdc4e328
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-04 22:00:58 +02:00
Matthias Sohn 1447159b92 JGit v4.4.0.201605041135-m1
Change-Id: Ic9b6d7dae293887c3ac3a11f92e153501abad1dc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-05-04 17:35:15 +02:00
Dave Borowitz c9b00288fd PersonIdent: Make appendSanitized public
Change-Id: I337c90fff3c013819a42a7622ac6bacc72828c07
2016-05-03 14:40:15 -07:00
Dave Borowitz 125a8959d2 PersonIdent: Strip some special chars from external strings
The special characters <> and '\n' interfere with parsing of
identities. C git strips these special characters, so we should too.

Rather than allocating extra strings by calling String#trim(), add a
few lines to our sanitization method to perform the same trimming as
described in String's Javadoc.

Change-Id: I96edcb93a2fc194ee354d60566d352299742a52f
2016-04-29 11:27:14 -04:00
Dave Borowitz 773f9661d0 PersonIdent: Document that name and email aren't trimmed
This might be somewhat surprising behavior to users who might
naturally assume the following invariant:
  ident.equals(parseIdent(ident.toExternalString()))

This invariant does not hold since whitespace is only trimmed during
serialization. We don't want to mess with the strings during
initialization, as this is called during the highly-optimized commit
parsing codepath.

Change-Id: I081a603f0ac0e33167462244779b0ff3ad51e80c
2016-04-29 11:27:06 -04:00
Dave Borowitz adff322a69 Expose the ObjectInserter that created an ObjectReader
We've found in Gerrit Code Review that it is common to pass around
both an ObjectReader (or more commonly a RevWalk wrapping one) and an
ObjectInserter. These code paths often assume that the ObjectReader
can read back any objects created by the ObjectInserter without
flushing. However, we previously had no way to enforce that constraint
programmatically, leading to hard-to-spot problems.

Provide a solution by exposing the ObjectInserter that created an
ObjectReader, when known. Callers can either continue passing both
objects and check:
  reader.getCreatedFromInserter() == inserter
or they can just pass around ObjectReader and extract the inserter
when it's needed (checking that it's not null at usage time).

Change-Id: Ibbf5d1968b506f6b47030ab1b046ffccb47352ea
2016-04-26 17:21:37 -04:00
Christian Halstrick 22d7ec2971 Fix calling of clean/smudge filters from Checkout,MergeCommands
When CheckoutCommand or MergeCommand is called then not in all situation
the treewalks have been prepared to support clean/smudge filters. Fix
this

Bug: 491505
Change-Id: Iab5608049221c46d06812552ab97299e44d59e64
2016-04-22 09:32:34 +02:00
Jon Schneider 80fa5f39f9 Fix ApplyCommand when result of patch is an empty file
Such hunks are identifiable by a zero value for "new start line". Prior
to the fix, JGit throws and ArrayIndexOutOfBoundsException on such
patches.

Change-Id: I4f3deb5e5f41a08af965fcc178d678c77270cddb
Signed-off-by: Jonathan Schneider <jkschneider@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-16 17:28:06 -05:00
Ned Twigg 32a5993227 Expose conflicting files in CheckoutConflictException
Change-Id: I5b3b7b0633354d5ccf0c6c320c0df9c93fdf8eeb
Signed-off-by: Ned Twigg <ned.twigg@diffplug.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-11 22:55:16 +02:00
Matthias Sohn eeeca8042f Merge "Fix CommitCommand to be able to skip writing to RefLog" 2016-04-11 16:50:11 -04:00
Christian Halstrick 36a53d1a3c Fix CommitCommand to be able to skip writing to RefLog
CommitCommand already provided a method to set the comment which should
be written into the reflog. The underlying RefUpdate class supported to
skip writing a reflog entry. But through the CommitCommand API it was
not possible to prevent writing a reflog entry. Fix this and allow
creating commits which don't occur in the reflog.

Change-Id: I193c53de71fb5958ea749c4bfa8360a51acc9b58
2016-04-11 08:58:48 +02:00
Matthias Sohn ccf8363693 Prepare 4.4.0-SNAPSHOT builds
Change-Id: If3162f4cc4ae6319b9f1e3293549485b039cfe7f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-08 17:59:59 +02:00
Matthias Sohn ac5d1bc37b Prepare 4.3.1-SNAPSHOT versions
Change-Id: I985828fd379e4596ae7f1c30a87e55df06a7924e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-08 17:24:30 +02:00
Matthias Sohn f2ee537d55 Prepare 4.3.1-SNAPSHOT versions
Change-Id: I52c98ba8fb3a303269a1f9380af114b6dd8c5009
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-08 00:56:43 +02:00
Matthias Sohn 4f0daa3bb2 JGit v4.3.0.201604071810-r
Change-Id: I902cdf1ff92ce8c6e9d80c4965d8d5bd8b9ac6c1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-08 00:10:10 +02:00
Matthias Sohn 57e6274bcf JGit v4.3.0.201604071045-r
Change-Id: Iafab78d6be34d31a13f979b7be67611135c0f8bd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-07 16:41:24 +02:00
Matthias Sohn 27d06cb8d5 Merge branch 'master' into stable-4.3
* master:
  Remove repository from cache when it's closed
  Fix RefDirectory not closing resources
  Fix repository cache never closing repository

Change-Id: I9dc9d017806cba25125f69b53812cc3e062ef975
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-07 13:06:23 +02:00
Hugo Arès 7d2b3b9e25 Remove repository from cache when it's closed
RepositoryCache has 2 methods to remove a repository from the cache but
they are never called when a repository is closed. Users of the cache
were expected to call one of those 2 methods but how could they have
called them at proper time without having visibility of the repository
usage count.

Ideally, I would have reworked the RepositoryCache to wrap any
repository it opens in a class that would be responsible to unregister
them from the cache when it's really closed, i.e. when usage counter
reaches 0. The problem preventing the wrapping solution is the
RepositoryCache.register method that allows to register an already
opened repository in the cache. Such repositories cannot be wrapped
because callers are still holding a reference on the unwrapped
repository.

Document that RepositoryCache.close method is removing the repository
from the cache as well as closing it and rework
RepositoryCache.unregister method to only remove the repository from the
cache. Use the latter to unregister repository when Repository.doClose
is getting executed.

Change-Id: Ia364816e4da8d7b6cfa72f10758ca31aa8a1f9db
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-07 13:00:50 +02:00
Hugo Arès 83235432e7 Fix repository cache never closing repository
Repository has a usage counter that is initialized to 1 at
instantiation and this counter is decremented when Repository.close
method is called. There is also a Repository.incrementOpen method that
RepositoryCache uses to increment the usage count when it's returning a
repository that is already opened.

The problem was that RepositoryCache was incrementing the usage count
for repositories that it just opened or registered. The usage count was
2 when it should have been 1.

Incrementing usage count is now only be done for repository that are
served from the cache.

This bug is causing slow memory increase of our Gerrit server until the
server become slow. Even if the RepositoryCache is using SoftReference,
it seems that the JVM is not garbage collecting the repositories because
it's not yet on the edge of being out of memory.

To test this change, I replicated all repositories(11k) from Gerrit
master to one slave. The Gerrit master used memory after this test was
10GB without this change and 3.5GB with.

Change-Id: I86c7b36174e384f106b51fe92f306018fd1dbdf0
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
2016-04-07 11:32:57 +02:00
Matthias Sohn f56b9001fc Merge branch 'master' into stable-4.3
* master:
  Add config parameter gc.prunePackExpire for packfile expiration
  In TestRepository, use a consistent clock

Change-Id: I7ac568e650fbd191e48a8f1a4068af72deb242e8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-06 16:05:14 +02:00
Christian Halstrick 47dd98e362 Enable calling of smudge filters when checking out paths
When checking out commits/branches JGit was triggering correctly
configured smudge filters. But when checking out paths (either from
index or from commits) JGit was not triggering smudge filters. Fix
CheckoutCommand to properly call filters.

Bug: 486560
Also-by: Pascal Krause <pascal.krausek@sap.com>
Change-Id: I5ff893054defe57ab12e201d901fe74e1376efea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-06 00:06:53 +02:00
Matthias Sohn 46bef2bcb6 Prepare 4.3-SNAPSHOT builds
Change-Id: Ib831f8870938113bd5338763f90a07d5c108b1de
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-04-05 23:10:06 +02:00
Preben Ingvaldsen 403f04d8dd Implement DIR_NO_GITLINKS
Implement the DIR_NO_GITLINKS setting with the same functionality
it provides in cGit.

Bug: 436200
Change-Id: I8304e42df2d7e8d7925f515805e075a92ff6ce28
Signed-off-by: Preben Ingvaldsen <preben@puppetlabs.com>
2016-04-05 10:43:40 +02:00