Commit Graph

1921 Commits

Author SHA1 Message Date
Thomas Wolf c27f36dfc7 Fix missing RefsChangedEvent when packed refs are used
With atomic ref updates using packed refs, JGit did not fire a
RefsChangedEvent. This resulted in a user-visible regression in
EGit: the UI would not update after a "Fetch from upstream...".
Presumably it would also make Gerrit miss out on ref changes?

Strengthen the BatchRefUpdateTest by also asserting the expected
number of RefsChangedEvents, and ensure modCnt is incremented in
RefDirectory.commitPackedRefs() when refs really changed (as opposed
to some internal housekeeping operation, such as packing loose refs).

Bug: 521296
Change-Id: Ia985bda1d99f45a5f89c8020ca4845e7a66e743e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-07 18:47:45 -04:00
Dave Borowitz bb09e09344 Add FetchCommand#setRefSpecs(String...) variant
Much of the time the caller can specify a RefSpec succinctly using a
string, and doesn't care about calling setters. Add a convenience method
for this case, and use it where applicable in JGit core.

Change-Id: Ic3fac7fc568eee4759236a5264d2e7e5f9b9716d
2017-09-07 07:46:25 -04:00
Shawn Pearce e68a9b3ed8 ReceivePack: clear advertised .haves if application changes refs
An application can choose to invoke setAdvertisedRefs multiple times,
for example several AdvertiseRefsHook installed in a chain. Each of
these invocations populates the advertisedHaves collection with the
unique set of ObjectIds.

This can lead to a server over-advertising with ".have" lines if the
first hook pushes in a lot of references, and the second hook filters
this to a subset.  ReceivePack will advertise the unique objects from
the first hook using ".have" lines, which may lead to a huge
advertisement sent to the client.

This can also contribute to a very slow connectivity check after the
pack is parsed as ReceivePack calls markUninteresting on every commit
in advertisedHaves.  This may require expanding a lot of subtrees to
mark all trees as uninteresting as well.  On a very big repository
this can lead to a many-second stall.

Clear the advertisedHaves collection any time the refs are updated.
Add a test to verify the correct set of objects was sent.

Change-Id: I97f6998d0597251444a2e846a3ea1f461bae96f9
2017-09-07 05:39:47 -04:00
David Pursehouse d6aec5da31 ObjectCheckerTest: Factor duplicate instances out to constants
The tests:

- testCheckBlobNotCorrupt
- testCheckBlobCorrupt

create instances of ObjectChecker that are the same.

The tests:

- testCheckBlobWithBlobObjectCheckerNotCorrupt
- testCheckBlobWithBlobObjectCheckerCorrupt

also create instances of ObjectChecker that are the same.

Factor these instances out to constants instead of creating them
in the tests.

The `checker` member is still created anew in each test, since some
of the tests change its state.

Change-Id: I2d90263829d01d208632185b1ec2f678ae1a3f4c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-06 00:02:28 +02:00
David Pursehouse 17e849145c SubmoduleConfigTest: Add additional tests in fetchRecurseToConfigValue
Add tests for "true" and "false" matching to "YES" and "NO".

Change-Id: I58223855022871ac4b21bd34ff6a9cd00fce30a1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-05 23:30:18 +02:00
Shawn Pearce d26309c4d9 Merge changes from topic 'dfs-reftable'
* changes:
  dfs: write reftable from DfsGarbageCollector
  dfs: compact reftables during DfsPackCompactor
2017-09-05 12:39:53 -04:00
Shawn Pearce d13dfac9dc dfs: write reftable from DfsGarbageCollector
If a ReftableConfig has been supplied by the caller, write out a
reftable as a sibling of the the GC pack, alongside the heads.

To bootstrap from a non-reftable system, the refs are read from the
DfsRefDatabase if no GC reftables are present.  Its assumed the
references are fully current, and do not need to be merged with any
other reftables.  Any non-GC reftables will be pruned at the end of
the GC cycle, just like any packs that were replaced.

If a GC reftable is present, all existing reftables are compacted, and
references from DfsRefDatabase are only used to seed the packer.  Its
assumed these are consistent with each other.

Change-Id: Ie397eb58aaaefb6865c816d9b39de3ac12998019
2017-09-05 09:10:16 -07:00
Thomas Wolf 11c476346d Fix Daemon.stop() to actually stop the listener thread
ServerSocket.accept() is not interruptible: a thread busy in accept()
may not react to Thread.interrupt() and may not return from accept()
via an InterruptedException. Close the socket instead to make the
daemon's listener thread terminate.

* Close the listening socket to get the listening thread to exit
  instead of interrupting it.
* Add a stopAndWait() method that stops the listening thread and
  then waits until it has indeed finished.
* Set SO_REUSE_ADDRESS on the listening socket.

Bug: 376369
Change-Id: I9d6014103e6dcb0173daea134feb44dc52c5c69a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-04 23:50:48 +02:00
Thomas Wolf 06ea633c18 Don't assume name = path in .gitmodules
While parsing .gitmodules, the name of the submodule subsection is
purely arbitrary: it frequently is the path of the submodule, but
there's no requirement for it to be. By building a map of paths to
the section name in .gitmodules, we can more accurately return
the submodule URL.

Bug: 508801
Change-Id: I8399ccada1834d4cc5d023344b97dcf8d5869b16
Also-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-04 09:19:03 +02:00
Thomas Wolf c91c20f36f Fix some tests for running in bazel
Some tests call out to external cgit. Those tests all failed for me
locally on Mac. Turned out that the reason was that the system git
config used by the git in the bazel run contained paths with ~/ but
somehow $HOME was not set. As a result the external git returned
with exit code 128.

Fix this by passing along $HOME explicitly. Also improve assertions
to make sure we do get the stderr of the external command in the
test log.

I hadn't noticed that until now because apparently the maven build
does pass along $HOME.

Change-Id: I7069676d5cc7b23a71e79a4866fe8acab5a405f4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-31 15:48:10 +02:00
Thomas Wolf c506f8d2dd Add dependency to jsch for OpenSshConfigTest to bazel build
Make jsch visible to the test bundle and add the dependency.

Change-Id: I0c49ee9b8f64fe8a8c74d2f08865917eb33069b4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-31 14:14:05 +02:00
Matthias Sohn e9fb111182 Cleanup: Organize imports
Change-Id: I6065e59519bc42bd18f5cc5ee3ec5210764ab03c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:33 +02:00
Matthias Sohn 41baa7a791 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Do not automatically organize imports using a save action since this
seems to be buggy and removed some annotations org.eclipse.jgit.pgm
needs to use args4j.

Change-Id: I5a91292c3b9241ce2dde3e4ecce14ad460097129
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:18 +02:00
Matthias Sohn 9906f09868 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Revert the following save actions which were introduced in c0ad77d8:
- always use braces around blocks
- remove unused imports

Other than I expected save actions are run globally on edited files -
and not only on edited code lines only.

Hence revert the save action "Convert control statement bodies to
blocks" which would affect a large number of code lines not affected by
the change editing some small part of a class. This would generate a
large number of changes which may lead to many unnecessary conflicts.
Total number of affected lines across jgit would be around 10k lines.

Also revert "Remove unused imports" since it erroneously removes imports
of some annotations needed by pgm classes using args4j.

Change-Id: I879a47f68e664129e6124cf25c1ae1f6a2d7a5aa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 01:24:47 +02:00
Shawn Pearce d684ade3d3 Merge "reftable: explicitly store update_index per ref" 2017-08-28 17:57:13 -04:00
David Pursehouse 10a8df22fa Merge "Enhance Eclipse save actions" 2017-08-28 12:08:01 -04:00
Terry Parker cb24de07d0 Merge "Add BlobObjectChecker" 2017-08-28 12:00:53 -04:00
Matthias Sohn c0ad77d84c Enhance Eclipse save actions
Add the following Eclipse save actions executed when saving modified
lines. This should help to reduce manual work needed to maintain a clean
and consistent code style:
- organize imports
- always use braces around blocks
- add missing annotations
  - @Override including implementation of interface methods
  - @Deprecated
- remove
  - unused imports
  - unnecessary $NON-NLS$ tags
  - redundant type arguments

Also add default values for new settings that were introduced in recent
Eclipse versions up to Neon since we updated save rules the last time.

Change-Id: Idc90b249df044d0552f04edf01a5f607c4846f50
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-28 11:52:45 -04:00
Masaya Suzuki fd74cf2f78 Add BlobObjectChecker
Some repositories can have a policy that do not accept certain blobs. To
check if the incoming pack file contains such blobs, ObjectChecker can
be used. However, this ObjectChecker is not called by PackParser if the
blob is stored as a whole. This is because the object can be so large
that it doesn't fit in memory.

This change introduces BlobObjectChecker. This interface takes chunks of
a blob instead of the entire object. ObjectChecker can optionally return
a BlobObjectChecker. This won't change existing ObjectChecker
implementation; existing implementation continues to receive deltified
blob objects only.

Change-Id: Ic33a92c2de42bd7a89786a4da26b7a648b25218d
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2017-08-28 08:42:27 -07:00
Thomas Wolf d031b64667 Exclude file matching: fix backtracking on match failures after **
** matching always tries the empty match first. If a mismatch occurs
later, the ** must be extended by exactly one segment and matching must
resume with the matcher following the ** matcher.

Bug: 520920
Change-Id: Id019ad1c773bd645ae92e398021952f8e961f45c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:41 +02:00
Thomas Wolf d80b999c76 Fix path pattern matching to work also for gitattributes
Path pattern matching for attribute rules is different than matching
for excluded files.

The first difference concerns patterns without slashes. For
gitattributes those must match on the last component only, not on
any earlier segment. This is true also for directory-only patterns.

The second difference concerns directory-only patterns. Those also
must not match on a prefix or segment except the last one. They do
not apply recursively to all files beneath.

And third, matches only on a prefix must match for gitattributes
only if the last matcher was "/**".

Add a new parameter for such path matching to IMatcher.matches() and
pass it through as appropriate (false for gitignore, true for
gitattributes). As far as gitignore is concerned, there is no change.

New tests have been added, and some existing attribute matching tests
have been fixed since they operated on wrong assumptions.

Bug: 508568
Change-Id: Ie825dc2cac8a85a72a7eeb0abb888f3193d21dd2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:40 +02:00
Thomas Wolf 08d4bef6b7 Add new tests for gitignore/gitattribute pattern matching
These tests verify that JGit matches the same as C git, for
both attribute matching (.gitattributes) and file exclusion matching
(.gitignore). These tests work by setting up a test repository and
test rules, and then determine excluded files or attributes both with
JGit and with the native C git, and then compare the results.

For .gitignore tests, we run

  git ls-files --ignored --exclude-standard -o
  
and for attribute tests we use

  git check-attr --stdin --all
  
and pass the list of all files in the repository via stdin.

Change-Id: I5b40946e04ff4a97456be7dffe09374323b7c89d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:40 +02:00
Matthias Sohn 7511bc886e Fix unthrown exception errors in CloneCommandTest
The Eclipse compiler raises errors for unthrown exceptions declared to
be thrown by test methods introduced in 88e45399.

Change-Id: I0d91c89e1b20ceff52c38b759abf906cc94e9902
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 09:11:02 +02:00
Matthias Sohn 960d7ff3e5 Prepare 4.5.4-SNAPSHOT builds
Change-Id: Id8b902bf2bf590b41f2e246c5ecf1592e1c411f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 08:08:46 +02:00
David Pursehouse 40f40e496a Merge "Fix default directory set when setDirectory wasn't called." 2017-08-25 20:57:52 -04:00
Thomas Wolf c758a8cd37 Do most %-token substitutions in OpenSshConfig
Except for %p and %r and partially %C, we can do token substitutions
as defined by OpenSSH inside the config file parser. %p and %r can
be replaced only if specified in the config; if not, it would be the
caller's responsibility to replace them with values obtained from the
URI to connect to.

Jsch doesn't know about token substitutions at all. By doing the
replacements as good as we can in the config file parser, we can
make Jsch support most of these tokens.

%i is not handled at all as Java has no concept of a "user ID".

Includes unit tests.

Bug: 496170
Change-Id: If9d324090707de5d50c740b0d4455aefa8db46ee
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 01:44:36 +02:00
Thomas Wolf 9d2447063d Let Jsch know about ~/.ssh/config
Ensure the Jsch instance used knows about ~/.ssh/config. This
enables Jsch to honor more user configurations (see
com.jcraft.jsch.Session.applyConfig()), in particular also the
UserKnownHostsFile configuration, or additional identities given
via multiple IdentityFile entries.

Turn JGit's OpenSshConfig into a full parser that can be a
Jsch-compliant ConfigRepository. This avoids a few bugs
in Jsch's OpenSSHConfig and keeps the JGit-facing interface
unchanged. At the same time we can supply a JGit OpenSshConfig
instance as a ConfigRepository to Jsch. And since they'll both
work from the same object, we can also be sure that the parsing
behavior is identical.

The parser does not handle the "Match" and "Include" keys, and it
doesn't do %-token substitutions (yet).

Note that Jsch doesn't handle multi-valued UserKnownHostFile
entries as known by modern OpenSSH.[1]

[1] http://man.openbsd.org/OpenBSD-current/man5/ssh_config.5

Additional tests for new features are provided in OpenSshConfigTest.

Bug: 490939
Change-Id: Ic683bd412fa8c5632142aebba4a07fad4c64c637
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 01:41:50 +02:00
Joan Goyeau 88e453995d Fix default directory set when setDirectory wasn't called.
Bug: 519883
Change-Id: I46716e9626b4c4adc0806a7c8df6914309040b94
Signed-off-by: Joan Goyeau <joan@goyeau.com>
2017-08-25 11:41:40 +01:00
Shawn Pearce 44a75d9ea8 reftable: explicitly store update_index per ref
Add an update_index to every reference in a reftable, storing the
exact transaction that last modified the reference.  This is necessary
to fix some merge race conditions.

Consider updates at T1, T3 are present in two reftables.  Compacting
these will create a table with range [T1,T3].  If T2 arrives during
or after the compaction its impossible for readers to know how to
merge the [T1,T3] table with the T2 table.

With an explicit update_index per reference, MergedReftable is able to
individually sort each reference, merging individual entries at T3
from [T1,T3] ahead of identically named entries appearing in T2.

Change-Id: Ie4065d4176a5a0207dcab9696ae05d086e042140
2017-08-21 15:39:08 -07:00
David Pursehouse 231f5d9baf Merge changes Id3994e2d,I5e2a2868,I255af794
* changes:
  LongObjectIdTest: Add back self comparison test
  Format BUILD files with buildifier
  Bazel: Add missing dependency in org.eclipse.jgit.http.test
2017-08-18 17:05:41 -04:00
David Pursehouse 904e9f8b73 Format BUILD files with buildifier
Change-Id: I5e2a286866b63a8fa2bd29cc2fe432fab2bbe0af
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-18 10:01:06 +01:00
Shawn Pearce 0aae64ce74 reftable: resolve symbolic references
resolve(Ref) helps callers recursively chase symbolic references and
is a useful function when wrapping a Reftable inside a RefDatabase, as
RefCursor does not resolve symbolic references during iteration.

Change-Id: I1ba143f403773497972e225dc92c35ecb989e154
2017-08-17 15:06:51 -07:00
Shawn Pearce d48ac5bf01 reftable: compact merged tables
A compaction of reftables is just copying the results of a
MergedReftable into a ReftableWriter.  Wrap this up into a utility.

Change-Id: I6f5677d923e9628993a2d8b4b007a9b8662c9045
2017-08-17 15:06:51 -07:00
Shawn Pearce 77d8eead6d reftable: merge-join reftables
MergedReftable combines multiple reference tables together in a stack,
allowing higher/later tables to shadow earlier/lower tables.  This
forms the basis of a transaction system, where each transaction writes
a new reftable containing only the modified references, and readers
perform a merge on the fly to get the latest value.

Change-Id: Ic2cb750141e8c61a8b2726b2eb95195acb6ddc83
2017-08-17 15:06:51 -07:00
Shawn Pearce 7da2fe6be8 reftable: lookup by ObjectId unit tests
Change-Id: Ic819a04e285094e271435dcd027d8006e5897785
2017-08-17 15:06:51 -07:00
Shawn Pearce e8e8041e83 reftable: reflog unit tests
Change-Id: If719a63ead54ecbcaf7cbe12c71f00435706bc2b
2017-08-17 15:06:51 -07:00
Shawn Pearce da5a27cd4e reftable: namespace unit tests
Add additional test cases for looking up entries within a namespace
such as refs/heads/ or refs/tags/, where the seek is passed a name
that ends with '/'.

Change-Id: I5f944de7518cd0090374bddba48d4dd3955a8d72
2017-08-17 15:06:51 -07:00
Shawn Pearce 9da26a5a1a reftable: bulk operation unit tests
Add more test cases that cover larger collections of
references, verifying every reference is accessible
both by scan and by seek.

Change-Id: Icada59fdcfc92a4634f6df61baaebb1c37b75d98
2017-08-17 15:06:51 -07:00
Shawn Pearce b6e25e5d7e reftable: basic functionality unit tests
This set of tests covers primitive storage of an empty
file, and each type of supported reference.

Change-Id: I3bdff35cae8ae27283051932f20608b3ac353559
2017-08-17 15:06:51 -07:00
Christian Halstrick be767fd7d9 Merge "Fix off-by-one error in Strings.count()" 2017-08-16 06:24:43 -04:00
Christian Halstrick c71af0c73a Merge "Use relative paths for attribute rule matching" 2017-08-16 06:24:33 -04:00
Matthias Sohn e21e2436d3 JGit v4.5.3.201708160445-r
Change-Id: I2d57144976e3683e180d3a42edc6c3bf2905e87c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-16 10:42:27 +02:00
Thomas Wolf b13a285098 Send a detailed event on working tree modifications
Currently there is no way to determine the precise changes done
to the working tree by a JGit command. Only the CheckoutCommand
actually provides access to the lists of modified, deleted, and
to-be-deleted files, but those lists may be inaccurate (since they
are determined up-front before the working tree is modified) if
the actual checkout then fails halfway through. Moreover, other
JGit commands that modify the working tree do not offer any way to
figure out which files were changed.

This poses problems for EGit, which may need to refresh parts of the
Eclipse workspace when JGit has done java.io file operations.

Provide the foundations for better file change tracking: the working
tree is modified exclusively in DirCacheCheckout. Make it emit a new
type of RepositoryEvent that lists all files that were modified or
deleted, even if the checkout failed halfway through. We update the
'updated' and 'removed' lists determined up-front in case of file
system problems to reflect the actual state of changes made.

EGit thus can register a listener for these events and then knows
exactly which parts of the Eclipse workspace may need to be refreshed.

Two commands manage checking out individual DirCacheEntries themselves:
checkout specific paths, and applying a stash with untracked files.
Make those two also emit such a new WorkingTreeModifiedEvent.

Furthermore, merges may modify files, and clean, rm, and stash create
may delete files.

CQ: 13969
Bug: 500106
Change-Id: I7a100aee315791fa1201f43bbad61fbae60b35cb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-15 16:52:00 -04:00
Matthias Sohn 81d020aba9 Merge branch 'stable-4.8'
* stable-4.8:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: Ica20f5aa0d8a365fe3317765b93520b3abd5d342
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-15 00:48:44 +02:00
Matthias Sohn 758a181b82 Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: I1e4fcf84506ff4316567bbb1713e84d8d196c2a1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-15 00:24:49 +02:00
Matthias Sohn 53becf1f59 Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: I669fe48ce0034f9ea1977d38ee39099497422c1c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 23:50:52 +02:00
Matthias Sohn 985e3c6414 Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
  Fix exception handling for opening bitmap index files

Change-Id: Ifb511238e3e98b1bc9f79a990807b940a17ebaa6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 23:43:05 +02:00
Christian Halstrick 1ed1e40387 Fix exception handling for opening bitmap index files
When creating a new PackFile instance it is specified whether this pack
has an associated bitmap index file or not. This information is cached
and the public method getBitmapIndex() will always assume a bitmap index
file must exist if the cached data tells so. But it may happen that the
packfiles are repacked during a gc in a different process causing the
packfile, bitmap-index and index file to be deleted. Since JGit still
has an open FileHandle on the packfile this file is not really deleted
and can still be accessed. But index and bitmap index file are deleted.
Fix getBitmapIndex() to invalidate the cached packfile instance if such
a situation occurs.

This problem showed up when a gerrit server was serving repositories
which where garbage collected with native git regularly. Fetch and
clone commands for certain repositories failed permanently after a
native git gc had deleted old bitmap index files.

Change-Id: I8e620bec74dd3f310ba42024f9a657062f868f0e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 21:09:48 +02:00
Thomas Wolf 37908321c0 Do not apply pushInsteadOf to existing pushUris
Per the git config documentation[1], pushInsteadOf is ignored when
a remote has explicit pushUris.

Implement this, and adapt tests.

Up to now JGit mistakenly applied pushInsteadOf also to existing
pushUris. If some repositories had relied on this mis-feature,
pushes may newly suddenly fail (the uncritical case; the config
just needs to be fixed) or even still succeed but push to unexpected
places, namely to the non-rewritten pushUrls (the critical case).

The release notes should point out this change.

[1] https://git-scm.com/docs/git-config

Bug: 393170
Change-Id: I38c83204d2ac74f88f3d22d0550bf5ff7ee86daf
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 17:27:05 +02:00
Thomas Wolf df3469f6ad Record submodule paths with untracked changes as FileMode.GITLINK
Bug: 520702
Change-Id: I9bb48af9e8f1f2ce7968a82297c7c16f1237f987
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 14:03:51 +02:00
Thomas Wolf f5a2c77dc4 Fix handling of pushInsteadOf
According to [1], pushInsteadOf is

1. applied to the uris, not to the pushUris
2. ignored if a remote has an explicit pushUri

JGit applied it only to the pushUris. As a result, pushInsteadOf was
ignored for remotes having only a uri, but no pushUri.

This commit implements (1) if there are no pushUris. I did not dare
implement (2) because:

* there are explicit tests for it that expect that pushInsteadOf gets
  applied to existing pushUrls, and
* people may actually use and rely on this JGit behavior.

[1] https://git-scm.com/docs/git-config

Bug: 393170
Change-Id: I6dacbf1768a105190c2a8c5272e7880c1c9c943a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 05:40:47 -04:00
Christian Halstrick 196915dde5 Merge "Ensure EOL stream type is DIRECT when -text attribute is present" 2017-08-14 03:34:57 -04:00
Thomas Wolf b07db60908 Fix off-by-one error in Strings.count()
Change-Id: I0667b1624827d1cf0cc1b81f86c7bb44eafd68a7
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 08:04:56 +02:00
Shawn Pearce 53dd9a9e4b Rename extensions.refsStorage to refStorage
This matches the proposal that has been discussed at length on
git-core mailing list and seems to be the accepted convention.

Change-Id: I9f6ab15144826893d1e2a4b48a2d657d6dd445ec
2017-08-11 18:20:50 -07:00
Thomas Wolf a489a8ae9a Ensure EOL stream type is DIRECT when -text attribute is present
Otherwise fancy combinations of attributes (binary or -text in
combination with crlf or eol) may result in the corruption of binary
data.

Bug: 520910
Change-Id: I3ffc666c13d1b9d2ed987b69a67bfc7f42ccdbfc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-11 22:56:50 +02:00
Thomas Wolf 4bc539a814 Use relative paths for attribute rule matching
Attribute rules must match against the entry path relative to the
attribute node containing the rule. The global entry path is to be
used only for the init and the global node (and of course the root
node).

Bug: 520677
Change-Id: I80389a2dc272a72312729ccd5358d7c75e1ea20a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-11 21:59:49 +02:00
Shawn Pearce ed29dec1ea Expose LongMap in util package
This is a useful primitive collection type like IntList.

Change-Id: I04b9b2ba25247df056eb3a1725602f1be6d3b440
2017-08-09 10:42:09 -07:00
Shawn Pearce 40c9c59e07 NB: encode and decode 24-bit ints
Change-Id: Ie036dc46e5a88a4e87dc52e880505bbe34601ca7
2017-08-09 10:42:09 -07:00
Matthias Sohn ba85764223 Remove unused import introduced in a551b64
Change-Id: Ia6c3935cf061590e7305d0a80a1051e9aebcbb43
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-01 23:26:44 +02:00
David Pursehouse 4085646f6d Merge changes I424295df,Ib003f7c8
* changes:
  Treat RawText of binary data as file with one single line.
  Trim boilerplate in RawParseUtils_LineMapTest.
2017-08-01 10:18:48 -04:00
Han-Wen Nienhuys a551b64694 Treat RawText of binary data as file with one single line.
This avoids executing mergeAlgorithm.merge on binary data, which is
unlikely to be useful.

Arguably, binary data should not make it to
ResolveMerger#contentMerge, but this approach has the following
advantages:

* binary detection is exact, since it doesn't only look at the start
  of the blob.

* it is cheap, as we have to iterate over the bytes anyway to find
  '\n'.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I424295df1dc60a719859d9d7c599067891b15792
2017-08-01 16:00:46 +02:00
Terry Parker 8c6a9a286e Merge "Use w1 for hashCode of AbbreviatedObjectId" 2017-07-28 19:24:11 -04:00
David Pursehouse 8391cc233b Merge "IntList: support contains(int)" 2017-07-28 14:18:21 -04:00
Shawn Pearce 4a00f18e8e Use w1 for hashCode of AbbreviatedObjectId
Very short abbreviations that are under 8 hex digits do not
have values in w2. Use w1 as the Java hashCode() instead, so
that the prefix of the abbreviation is always included in the
hashing function used by any java.util.Collection type.

Change-Id: Idaf69f86b62630ba4a022d31b4c293c6d138f557
2017-07-28 10:20:45 -07:00
Shawn Pearce 652a6b0334 IntList: support contains(int)
LongList supports contains(long).
IntList should also support contains(int).

Change-Id: Ic7a81c3c25b0f10d92087b56e9f200b676060f63
2017-07-28 10:18:27 -07:00
Dave Borowitz 45da0fc6f7 RefDirectory: Add in-process fair lock for atomic updates
In a server scenario such as Gerrit Code Review, there may be many
atomic BatchRefUpdates contending for locks on both the packed-refs file
and some subset of loose refs. We already retry lock acquisition to
improve this situation slightly, but we can do better by using an
in-process lock. This way, instead of retrying and potentially exceeding
their timeout, different threads sharing the same Repository instance
can wait on a fair lock without having to touch the disk lock. Since a
server is probably already using RepositoryCache anyway, there is a high
likelihood of reusing the Repository instance.

Change-Id: If5dd1dc58f0ce62f26131fd5965a0e21a80e8bd3
2017-07-28 11:03:32 -04:00
Dave Borowitz 6f23210781 RefDirectory: Retry acquiring ref locks with backoff
If a repo frequently uses PackedBatchRefUpdates, there is likely to be
contention on the packed-refs file, so it's not appropriate to fail
immediately the first time we fail to acquire a lock. Add some logic to
RefDirectory to support general retrying of lock acquisition.

Currently, there is a hard-coded wait starting at 100ms and backing off
exponentially to 1600ms, for about 3s of total wait. This is no worse
than the hard-coded backoff that JGit does elsewhere, e.g. in
FileUtils#delete. One can imagine a scheme that uses per-repository
configuration of backoff, and the current interface would support this
without changing any callers.

Change-Id: I4764e11270d9336882483eb698f67a78a401c251
2017-07-28 07:53:25 -04:00
David Pursehouse 5188c23104 Merge "Fix committing empty commits" 2017-07-28 06:08:33 -04:00
David Pursehouse 94aebcb949 Merge "Support overriding a batch's reflog on a per-ReceiveCommand basis" 2017-07-28 06:07:08 -04:00
Christian Halstrick da0770fdec Fix committing empty commits
Allow to explicitly create an empty commit even if committing only
certain files.

Bug: 510685 
Change-Id: If9bf664d7cd824f8e5bd6765fa6cc739af3d7721
2017-07-28 10:46:42 +01:00
David Pursehouse 7e4946626e Merge changes from topic 'batch-ref-update-reflog'
* changes:
  BatchRefUpdate: Expand javadocs and add @Nullable
  PackedBatchRefUpdate: Write reflogs
  Extract constants for reflog entry message prefixes
2017-07-28 05:40:45 -04:00
Han-Wen Nienhuys ab0eedcead Trim boilerplate in RawParseUtils_LineMapTest.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Ib003f7c8f2816dd57e941799a665e70ecd6645a2
2017-07-27 13:49:39 +02:00
Zhen Chen b0695e5b7b Add commit check for head references
Make sure all refs/heads/* point to a commit object.

Change-Id: I9c7cf347aaf63d5ef604d520c2383c6cf3043890
Signed-off-by: Zhen Chen <czhen@google.com>
2017-07-26 10:12:37 -07:00
Zhen Chen 673acfc6bd Add connectivity check from references
Make sure all objects referenced by references are reachable. Stop at
the first missing object.

Change-Id: Ifcd7392c4321b17d9290bd87f038bc62bc10dabb
Signed-off-by: Zhen Chen <czhen@google.com>
2017-07-26 10:12:37 -07:00
Zhen Chen 2c2999643f Add dfs fsck implementation
JGit already had some fsck-like classes like ObjectChecker which can
check for an individual object.

The read-only FsckPackParser which will parse all objects within a pack
file and check it with ObjectChecker. It will also check the pack index
file against the object information from the pack parser.

Change-Id: Ifd8e0d28eb68ff0b8edd2b51b2fa3a50a544c855
Signed-off-by: Zhen Chen <czhen@google.com>
2017-07-26 10:12:29 -07:00
Dave Borowitz 104107bf43 Support overriding a batch's reflog on a per-ReceiveCommand basis
Change-Id: I86a4b8f6b4f85b2bae64c1b121e4ee527d46de83
2017-07-26 11:40:15 -04:00
Dave Borowitz 22e9106224 PackedBatchRefUpdate: Write reflogs
On-disk reflogs are not stored in the packed-refs file, so we cannot
ensure atomic updates. We choose the lesser evil of dropping failed
reflog updates on the floor, rather than throwing an exception even
though the underlying ref updates succeeded.

Add tests for reflogs to BatchRefUpdateTest.

Change-Id: Ia456ba9e36af8e01fde81b19af46a72378e614cd
2017-07-26 11:39:33 -04:00
Dave Borowitz 82f68500c0 Improve BatchRefUpdateTest readability
* Factor out helpers for setting up and executing updates.
* Use common assert methods, with a special enum type that papers over
  the fact that there is no ReceiveCommand.Result for transaction
  aborted.
* Static import ReceiveCommand.Type constants.
* Add blank lines to separate repo setup, update execution, and asserts.

Change-Id: Ic3717f94331abfc7ae3e92065f3fe32026bf7cea
2017-07-25 13:14:50 -04:00
Dave Borowitz 65b64768b3 Move BatchRefUpdate tests to a new file
Run with @Parameterized, so we don't have to duplicate test setup for
each atomic/non-atomic test. We still have to have two different sets of
asserts for the cases where the behavior is different. In fact, this is
a readability win: it emphasizes that performing the exact same setup
except for the atomic setting will have different behavior.

Change-Id: I78a8214075e204732a423341f14c09de273a7854
2017-07-25 13:14:50 -04:00
Dave Borowitz 26962861d4 Implement atomic BatchRefUpdates for RefDirectory
The existing packed-refs file provides a mechanism for implementing
atomic multi-ref updates without any changes to the on-disk format or
lockfile protocol. We just need to make sure that there are no loose
refs involved in the transaction, which we can achieve by packing the
refs while holding locks on all loose refs. Full details of the
algorithm are in the PackedBatchRefUpdate javadoc.

This change does not implement reflog support, which will come in a
later change.

Change-Id: I09829544a0d4e8dbb141d28c748c3b96ef66fee1
2017-07-25 13:14:50 -04:00
Dave Borowitz cf9e3fad52 Separate RefUpdate.Result.REJECTED_{MISSING_OBJECT,OTHER_REASON}
ReceiveCommand.Result has a slightly richer set of possibilities, so it
makes sense for RefUpdate.Result to have more values in order to match.
In particular, this allows us to return REJECTED_MISSING_OBJECT from
RefUpdate when an object is missing.

The comment in RefUpdate#safeParse about expecting some old objects to be
missing is only applicable to the old ID, not the new ID. A missing new
ID is a bug or programmer error, and we should not update a ref to point
to one.

Fix various tests that started failing because they depended for no good
reason on setting refs to point to nonexistent objects; it's always easy
to create a real object when necessary.

It is possible that some downstream users of RefUpdate.Result might
choose to handle one of the new statuses differently, for example by
providing a more user-readable error message; that is not done in this
change.

Change-Id: I734b1c32d5404752447d9e20329471436ffe05fc
2017-07-25 13:12:34 -04:00
David Pursehouse 4940ea14b7 Add missing newlines at ends of Java files
Change-Id: Iead36f53d57ead0eb3edd3f9efb63b6630c9c20c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-07-25 10:37:21 +01:00
Dave Borowitz e93b0026ce Temporarily @Ignore flaky CommitCommandTest methods
Change-Id: Ia2c42d014323bd29b85bf76f1a20c83f612406d7
2017-07-25 05:17:12 -04:00
Dmitry Pavlenko 843e444561 Fix matching ignores and attributes pattern of form a/b/**.
Fix patch matching for patterns of form a/b/** : this should not match
paths like a/b but still match a/b/ and a/b/c.

Change-Id: Iacbf496a43f01312e7d9052f29c3f9c33807c85d
Signed-off-by: Dmitry Pavlenko <pavlenko@tmatesoft.com>
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-07-24 09:16:33 +01:00
David Pursehouse ba91e8a086 Merge changes from topic 'packed-batch-ref-update'
* changes:
  Add tests for updating single refs to missing objects
  Fix deleting symrefs
  RefDirectory: Throw exception if CAS of packed ref list fails
  ReceiveCommand: Explicitly check constructor preconditions
  BatchRefUpdate: Document when getPushOptions is null
2017-07-24 03:38:42 -04:00
Shawn Pearce a1e4825c4e dfs: silence resource warnings in DfsBlockCacheTest
Change-Id: Ia934d8578592dc20837944d50acfb8920e260893
2017-07-19 14:59:50 -07:00
Shawn Pearce f414f7de1f dfs: Fix DataFormatException: 0 bytes to inflate
When a file uses a different block size (e.g.  500) than the cache
(e.g.  512), and the DfsPackFile's blockSize field has not been
initialized, the cache misaligns block loads.  The cache uses its
default of 512 to compute the block alignment instead of the file's
500.

This causes DfsReader try to set an empty range into an Inflater,
resulting in an object being unable to load.

Change-Id: I7d6352708225f62ef2f216d1ddcbaa64be113df6
2017-07-19 14:28:59 -07:00
Shawn Pearce 3a072c8a7a dfs: test for repositories sharing blocks in DfsBlockCache
Simple test to verify two DfsRepository instances will reuse the same
DfsBlocks in the DfsBlockCache, even though the DfsStreamKey instance
is now different between their DfsPackFile instances.

Change-Id: I409c109142dea488d189b9ac0d3c319755dce7b4
2017-07-19 13:48:17 -07:00
Shawn Pearce 8d27c480df dfs: Fix build break caused by DfsStreamKey.of signature change
Change-Id: I6c49cf42a04dd0d96cfe0751f500a51f56f0bdb8
2017-07-19 09:32:00 -07:00
Shawn Pearce 07f98a8b71 Derive DfsStreamKey from DfsPackDescription
By making this a deterministic function, DfsBlockCache can stop
retaining a map of every DfsPackDescription it has ever seen.  This
fixes a long standing memory leak in DfsBlockCache.

This refactoring also simplifies the idea of setting up more
lightweight objects around streams.

Change-Id: I051e7b96f5454c6b0a0e652d8f4a69c0bed7f6f4
2017-07-17 13:20:34 -07:00
Dave Borowitz 28202a6758 Add tests for updating single refs to missing objects
The reader may find it surprising that this succeeds without incident
unless there is peeling or a fast-forward check involved. This behavior
may be changed in the future, but for now, just document the current
behavior.

Change-Id: I348b37e93e0264dc0905c4d58ce881852d1dfe5e
2017-07-17 11:56:35 -04:00
Dave Borowitz f529fa6729 Fix deleting symrefs
The RefDirectory implementation of doDelete never considered whether to
delete a symref or its leaf, because the detachingSymbolicRef bit was
never exposed from RefUpdate. The behavior was thus incorrectly to
always delete the symref, never the leaf.

There was no test for this behavior. The only thing that attempted to be
a test was testDeleteHeadInBareRepo, but this test was broken for
reasons unrelated to this bug. Specifically, it set the leaf to point to
a completely nonexistent object, and then asserted that deleting HEAD
resulted in NO_CHANGE. The only reason this test ever passed is because
of a quirk of updateImpl, which treats a missing object as the same as
null. This quirk aside, the test wasn't really testing the right thing.
Turn this into a real test by writing out a real object and pointing the
leaf at that.

Also, add a test for the detachingSymbolicRef case, i.e. deleting the
symref and leaving the leaf alone.

Change-Id: Ib96d2a35b4f99eba0734725486085fc6f9d78aa5
2017-07-17 11:56:35 -04:00
Dave Borowitz 9c33f7364d RefDirectory: Throw exception if CAS of packed ref list fails
The contents of the packedRefList AtomicReference should never differ
from what we expect prior to writing, because this segment of the code
is protected by the packed-refs lock file on disk. If it does happen,
whether due to programmer error or a rogue process not respecting the
locking protocol, it's better to let the caller know than to silently
drop the whole commit operation on the floor.

The existing concurrentOnlyOneWritesPackedRefs test is inherently
nondeterministic as written, and was already about 6% flaky as measured
by bazel:

  $ bazel test --runs_per_test=200 //org.eclipse.jgit.test:org_eclipse_jgit_internal_storage_file_GcPackRefsTest
  ...
  INFO: Elapsed time: 42.608s, Critical Path: 10.35s
  //org.eclipse.jgit.test:org_eclipse_jgit_internal_storage_file_GcPackRefsTest FAILED in 12 out of 200 in 1.6s
    Stats over 200 runs: max = 1.6s, min = 1.1s, avg = 1.3s, dev = 0.1s

This flakiness was caused by the assumption that exactly one of the 2
threads would fail, when both might actually succeed in practice due to
racing on the compare-and-swap.

For whatever reason, this change affected the interleaving behavior in
such a way that the flakiness jumped to around 50%. Making the
interleaving of the test fully deterministic is beyond the scope of this
change, but a simple tweak to the assertion is enough to make it pass
consistently 200+ times both before and after this change.

Change-Id: I5ff4dc39ee05bda88d47909acb70118f3d0c8f74
2017-07-17 11:56:35 -04:00
Dave Borowitz 21ec281f3e ReceiveCommand: Explicitly check constructor preconditions
Some downstream code checks whether a ReceiveCommand is a create or a
delete based on the type field. Other downstream code (in particular a
good chunk of Gerrit code I wrote) checks the same thing by comparing
oldId/newId to zeroId. Unfortunately, there were no strict checks in the
constructor that ensures that zeroId is only set for oldId/newId if the
type argument corresponds, so a caller that passed mismatched IDs and
types would observe completely undefined behavior as a result. This is
and always has been a misuse of the API; throw IllegalArgumentException
so the caller knows that it is a misuse.

Similarly, throw from the constructor if oldId/newId are null. The
non-nullness requirement was already documented. Fix RefDirectoryTest to
not do the wrong thing.

Change-Id: Ie2d0bfed8a2d89e807a41925d548f0f0ce243ecf
2017-07-17 11:56:35 -04:00
Shawn Pearce e924de5295 Rename DfsPackKey to DfsStreamKey
This renaming supports reusing DfsStreamKey in a future commit
to index other PackExt type streams inside of the DfsBlockCache.

Change-Id: Ib52d374e47724ccb837f4fbab1fc85c486c5b408
2017-07-17 08:15:37 -07:00
Christian Halstrick 1968b20066 Merge "Support -merge attribute in binary macro" 2017-07-03 07:48:19 -04:00
Mathieu Cartaud f7e233e450 Support -merge attribute in binary macro
The merger is now able to react to the use of the merge attribute.
The value unset and the custom value 'binary' are handled (-merge
and merge=binary)

Since the specification of the merge attribute states that when the
attribute is unset, ours version must be kept in case of a conflict, we
don't overwrite the file but keep the local version.

Bug: 517128
Change-Id: Ib5fbf17bdaf727bc5d0e106ce88f2620d9f87a6f
Signed-off-by: Mathieu Cartaud <mathieu.cartaud@obeo.fr>
2017-06-27 10:33:50 +02:00
David Turner 695e38a83b Add a test for parsing fsck config options and expose FsckMode enum
These config options allow overriding the message type (error, warn or
ignore) of a specific message ID such as missingEmail.
The supported fsck message IDs are defined in ObjectChecker.ErrorType.

Since TransferConfig.FsckMode wasn't public parsing fsck configuration
options like e.g. fsck.missingEmail=ignore failed with an
IllegalAccessException. Fix this by declaring this enum public.

Change-Id: I3f41ff7a76a846250a63ce92a9fd111eb347269f
Signed-off-by: David Turner <dturner@twosigma.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-06-23 00:10:20 +02:00
Oliver Lockwood 060f3699d4 Fix bug in multiple tag handling on DescribeCommand
In the case of multiple tags on the same commit, jgit previously
only ever looked at the last of those tags; git behaviour is to
return the first tag (or first matching one if --match is
specified).

Bug: 518377
Change-Id: I3b6b58ad9f8aa3879ae35b84542b7bddc74a27d6
Signed-off-by: Oliver Lockwood <oliver.lockwood@cantab.net>
2017-06-21 17:25:19 +01:00
Oliver Lockwood af0867cb86 Support --match functionality in DescribeCommand
A `match()` method has been added to the DescribeCommand, allowing
users to specify one or more `glob(7)` matchers as per Git convention.

Bug: 518377
Change-Id: Ib4cf34ce58128eed0334adf6c4a052dbea62c601
Signed-off-by: Oliver Lockwood <oliver.lockwood@cantab.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-06-20 00:23:26 +02:00