Commit Graph

3877 Commits

Author SHA1 Message Date
David Pursehouse de5eb5e2cd FsckPackParser: Fix "Empty block should be documented" warnings
Add "Do nothing" comments, consistent with other empty methods in
the same class.

Change-Id: I27a13a402e94104af617be0e14d8982e75fa73bd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
David Pursehouse 9c1fcf6ad7 FsckPackParser: Fix warnings about variable hiding
Change-Id: Id1ada1cedfad5671e79df303c6dbcb6017b777ce
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
David Pursehouse f8c2067410 Silence API errors caused by adding enum constants in 1baf86d4d2
Change-Id: I23338c04567f95aedd909ab310e01277e8f7db69
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
Shawn Pearce f4329b09d4 Merge changes from topic 'dfs-reftable'
* changes:
  dfs: helper to open multiple reftables
  dfs: expose DfsReftable from DfsObjDatabase
  dfs: support reading reftables through DfsBlockCache
2017-08-30 20:42:23 -04:00
Shawn Pearce 0ca1fb0fe0 Merge changes from topic 'fsck'
* changes:
  DfsFsck: reduce memory usage during verifyIndex
  DfsFsck: refactor pack verify into its own method
  DfsFsck: run connectivity check pass exactly once
2017-08-30 08:42:37 -04:00
Shawn Pearce e5db7c1f0e DfsFsck: reduce memory usage during verifyIndex
Don't convert a lot of ObjectId to String stored in generic
java.util.HashSet.  This is a very expensive way to store objects.

Instead rely on "this" from the FsckPackParser to lookup information
about the objects in this pack file, which lets the verify code avoid
sorting the object list.

Use ObjectIdOwnerMap, which is the most efficient format JGit has
for storing lots of objects.

Change-Id: Ib68f93acb4d91b96d0a44c0612f704500d332ac1
2017-08-30 03:11:53 -07:00
Shawn Pearce 69588c21fe DfsFsck: refactor pack verify into its own method
This simplifies the logic about allocation of the DfsReader, and
clarifies the code considerably by using smaller scopes with less
indentation.

A few static imports from PackExt and slightly shorter variable names
make for a more understandable-at-glance implementation.

Change-Id: Iaf5a0e14fe0349215d9e44446f68d1129ad3bb3d
2017-08-30 03:11:43 -07:00
Shawn Pearce 153c11a49b DfsFsck: run connectivity check pass exactly once
The simpler algorithm is to load all branch tips into an ObjectWalk
and run that walk exactly once.  This avoids redoing work related to
parsing and considering trees reused across side branches.

Move the connectivity check into its own helper method.  This moves it
left one level of identation, and makes it easier to fit the method's
logic with less line wrapping.

Add a "Counting objects..." progress monitor around this phase.  Its
what is used when a server receives a push and is also trying to
verify the client sent all required objects.

Change-Id: I4d53d75d0cdd1a13fff7d513a6ae0b2d14ea4090
2017-08-29 18:14:51 -07:00
Matthias Sohn f4c5a54aa9 Cleanup: Remove redundant type arguments
Change-Id: I81ce2e25666f5acfce4d3ada6b71ae8c81a97a4d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:34 +02:00
Matthias Sohn e9fb111182 Cleanup: Organize imports
Change-Id: I6065e59519bc42bd18f5cc5ee3ec5210764ab03c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:33 +02:00
Matthias Sohn 41baa7a791 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Do not automatically organize imports using a save action since this
seems to be buggy and removed some annotations org.eclipse.jgit.pgm
needs to use args4j.

Change-Id: I5a91292c3b9241ce2dde3e4ecce14ad460097129
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:18 +02:00
Matthias Sohn 1052678372 Cleanup: Remove trailing whitespace
Change-Id: I7c08b60fb6afafd1874210320da3da7519632e08
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 01:24:48 +02:00
Matthias Sohn 9906f09868 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Revert the following save actions which were introduced in c0ad77d8:
- always use braces around blocks
- remove unused imports

Other than I expected save actions are run globally on edited files -
and not only on edited code lines only.

Hence revert the save action "Convert control statement bodies to
blocks" which would affect a large number of code lines not affected by
the change editing some small part of a class. This would generate a
large number of changes which may lead to many unnecessary conflicts.
Total number of affected lines across jgit would be around 10k lines.

Also revert "Remove unused imports" since it erroneously removes imports
of some annotations needed by pgm classes using args4j.

Change-Id: I879a47f68e664129e6124cf25c1ae1f6a2d7a5aa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 01:24:47 +02:00
Thomas Wolf dbef8e2537 Pass along the original exception when an ssh connection fails
Otherwise, the stack trace doesn't really tell anything.
See for instance [1].

[1] https://www.eclipse.org/forums/index.php/t/1088535/

Change-Id: If22f2c63c36fec6b32818d2c2acecf20531b4185
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-29 15:08:08 -04:00
Shawn Pearce 4f7ce2b46b dfs: helper to open multiple reftables
ReftableStack maintains multiple open reftables
in an AutoCloseable format, making it easier for
higher level code to handle multiple files.

Change-Id: I7ac35c18e67b7e771fb3de29169d1ee50fab62ca
2017-08-28 15:07:05 -07:00
Shawn Pearce 1a7b8a11df dfs: expose DfsReftable from DfsObjDatabase
Reftable storage in DFS is related to pack storage.  Reftables are
stored in the same namespace, but with PackExt.REFTABLE.  Include
the set of DfsReftable instances in the PackList and export some
helpers to access the tables.

Change-Id: I6a4f5f953ed6b0ff80a7780f4c6cbcc5eda0da3e
2017-08-28 15:07:04 -07:00
Shawn Pearce 1222f34506 dfs: support reading reftables through DfsBlockCache
DfsBlockCache directly shares its internal byte[] with ReftableReader,
avoding copying between the DfsBlockCache and the BlockReader
instances used by ReftableReader.

Change-Id: Icaa4f40052b26f952681414653a8b5314b7c2c23
2017-08-28 15:07:04 -07:00
Shawn Pearce d684ade3d3 Merge "reftable: explicitly store update_index per ref" 2017-08-28 17:57:13 -04:00
David Pursehouse 10a8df22fa Merge "Enhance Eclipse save actions" 2017-08-28 12:08:01 -04:00
Terry Parker cb24de07d0 Merge "Add BlobObjectChecker" 2017-08-28 12:00:53 -04:00
Matthias Sohn c0ad77d84c Enhance Eclipse save actions
Add the following Eclipse save actions executed when saving modified
lines. This should help to reduce manual work needed to maintain a clean
and consistent code style:
- organize imports
- always use braces around blocks
- add missing annotations
  - @Override including implementation of interface methods
  - @Deprecated
- remove
  - unused imports
  - unnecessary $NON-NLS$ tags
  - redundant type arguments

Also add default values for new settings that were introduced in recent
Eclipse versions up to Neon since we updated save rules the last time.

Change-Id: Idc90b249df044d0552f04edf01a5f607c4846f50
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-28 11:52:45 -04:00
Masaya Suzuki fd74cf2f78 Add BlobObjectChecker
Some repositories can have a policy that do not accept certain blobs. To
check if the incoming pack file contains such blobs, ObjectChecker can
be used. However, this ObjectChecker is not called by PackParser if the
blob is stored as a whole. This is because the object can be so large
that it doesn't fit in memory.

This change introduces BlobObjectChecker. This interface takes chunks of
a blob instead of the entire object. ObjectChecker can optionally return
a BlobObjectChecker. This won't change existing ObjectChecker
implementation; existing implementation continues to receive deltified
blob objects only.

Change-Id: Ic33a92c2de42bd7a89786a4da26b7a648b25218d
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2017-08-28 08:42:27 -07:00
Thomas Wolf 1637c44048 FetchCommand: pass on CredentialsProvider to submodule fetches
When a JGit API command is implemented in terms of other API
commands, the child command must "inherit" all relevant settings.
Calling configure() ensures that the CredentialsProvider and the
connection timeout are propagated correctly.

Bug: 515325
Change-Id: I948e306693a9edb7b199a735877413b6eddcfba4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:37:43 +02:00
Thomas Wolf d031b64667 Exclude file matching: fix backtracking on match failures after **
** matching always tries the empty match first. If a mismatch occurs
later, the ** must be extended by exactly one segment and matching must
resume with the matcher following the ** matcher.

Bug: 520920
Change-Id: Id019ad1c773bd645ae92e398021952f8e961f45c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:41 +02:00
Thomas Wolf d80b999c76 Fix path pattern matching to work also for gitattributes
Path pattern matching for attribute rules is different than matching
for excluded files.

The first difference concerns patterns without slashes. For
gitattributes those must match on the last component only, not on
any earlier segment. This is true also for directory-only patterns.

The second difference concerns directory-only patterns. Those also
must not match on a prefix or segment except the last one. They do
not apply recursively to all files beneath.

And third, matches only on a prefix must match for gitattributes
only if the last matcher was "/**".

Add a new parameter for such path matching to IMatcher.matches() and
pass it through as appropriate (false for gitignore, true for
gitattributes). As far as gitignore is concerned, there is no change.

New tests have been added, and some existing attribute matching tests
have been fixed since they operated on wrong assumptions.

Bug: 508568
Change-Id: Ie825dc2cac8a85a72a7eeb0abb888f3193d21dd2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:40 +02:00
Thomas Wolf 426caf99ee Ignore invalid TagOpt values
C git silently ignores invalid tagopt values; so make JGit behave the
same way.

Bug: 429625
Change-Id: I99587cc46c7e0c19348bcc63f602038fa9a7f378
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 09:11:03 +02:00
Thomas Wolf 8cbdf523cd Add a getter for a list of RefSpecs to Config
Reading RefSpecs from a Config can be seen as another typed value
conversion, so add a getter to Config and to TypedConfigGetter. Use
it in RemoteConfig.

Doing this allows clients of the JGit library to customize the
handling of invalid RefSpecs in git config files by installing a
custom TypedConfigGetter.

Bug: 517314
Change-Id: I0ebc0f073fabc85c2a693b43f5ba5962d8a795ff
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 09:11:02 +02:00
Thomas Wolf d32ad1cadd Improve getting typed values from a Config
Make the handling of typed values somewhat configurable by using
a separate converter. The default converter is the same as before;
just the implementations of the getters were moved. They also still
raise IllegalArgumentException on invalid values as before.

The converter can be set globally via Config.setTypedConfigGetter(),
which EGit can use in its core Activator to plug in a variant that
catches the IllegalArgumentException, logs the problem, and then
returns the default value.

In this way the behavior for other users of the JGit library is
unchanged, while EGit can deal gracefully with invalid git configs.

Bug: 520978
Change-Id: Ie8f81d206e358b6cc57aa29b9d7ad2a5d34b86a1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 09:11:02 +02:00
David Pursehouse e237c28936 Merge "Fix JGit set core.fileMode to false by default instead of true for non Windows OS." 2017-08-25 20:58:07 -04:00
David Pursehouse 40f40e496a Merge "Fix default directory set when setDirectory wasn't called." 2017-08-25 20:57:52 -04:00
David Pursehouse 0e12692d8c FileMode: Remove unnecessary @SuppressWarnings("synthetic-access")
In Eclipse Oxygen, the following warning is emitted:

  At least one of the problems in category 'synthetic-access' is not
  analysed due to a compiler option being ignored

Removing the suppression gets rid of the warning.

Change-Id: Ibfe5cc1e347150b699f54e2f204ab5ee770da202
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-25 20:57:11 -04:00
Matthias Sohn d979dfd00c Add toString() methods to OpenSshConfig to help debugging
Change-Id: I81b60a13a97e78d5ccd593ba8e4aa614df19f925
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 01:44:37 +02:00
Thomas Wolf c758a8cd37 Do most %-token substitutions in OpenSshConfig
Except for %p and %r and partially %C, we can do token substitutions
as defined by OpenSSH inside the config file parser. %p and %r can
be replaced only if specified in the config; if not, it would be the
caller's responsibility to replace them with values obtained from the
URI to connect to.

Jsch doesn't know about token substitutions at all. By doing the
replacements as good as we can in the config file parser, we can
make Jsch support most of these tokens.

%i is not handled at all as Java has no concept of a "user ID".

Includes unit tests.

Bug: 496170
Change-Id: If9d324090707de5d50c740b0d4455aefa8db46ee
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 01:44:36 +02:00
Thomas Wolf 9d2447063d Let Jsch know about ~/.ssh/config
Ensure the Jsch instance used knows about ~/.ssh/config. This
enables Jsch to honor more user configurations (see
com.jcraft.jsch.Session.applyConfig()), in particular also the
UserKnownHostsFile configuration, or additional identities given
via multiple IdentityFile entries.

Turn JGit's OpenSshConfig into a full parser that can be a
Jsch-compliant ConfigRepository. This avoids a few bugs
in Jsch's OpenSSHConfig and keeps the JGit-facing interface
unchanged. At the same time we can supply a JGit OpenSshConfig
instance as a ConfigRepository to Jsch. And since they'll both
work from the same object, we can also be sure that the parsing
behavior is identical.

The parser does not handle the "Match" and "Include" keys, and it
doesn't do %-token substitutions (yet).

Note that Jsch doesn't handle multi-valued UserKnownHostFile
entries as known by modern OpenSSH.[1]

[1] http://man.openbsd.org/OpenBSD-current/man5/ssh_config.5

Additional tests for new features are provided in OpenSshConfigTest.

Bug: 490939
Change-Id: Ic683bd412fa8c5632142aebba4a07fad4c64c637
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 01:41:50 +02:00
Masaya Suzuki 9fb6561e7a Consume request body before flushing the buffer
This is continuation from https://git.eclipse.org/r/#/c/94249/. When an
error happens, we might not read the entire stream. Consume the request
body before we flush the buffer.

Change-Id: Ia473a04ace600653b2d1f2822e3023570d992410
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2017-08-25 15:23:20 -07:00
Joan Goyeau 88e453995d Fix default directory set when setDirectory wasn't called.
Bug: 519883
Change-Id: I46716e9626b4c4adc0806a7c8df6914309040b94
Signed-off-by: Joan Goyeau <joan@goyeau.com>
2017-08-25 11:41:40 +01:00
David Pursehouse 65b2d0b2d9 ObjectToPack: Add missing @Override annotation
Change-Id: I65ed7b89312d58ea816b46d27707ff907df1c78b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-24 16:20:11 +09:00
Thomas Wolf 1b4daa2994 Cleanup: message reporting for HTTP redirect handling
The addition of "tooManyRedirects" in commit 7ac1bfc ("Do
authentication re-tries on HTTP POST") was an error I didn't
catch after rebasing that change. That message had been renamed
in the earlier commit e17bfc9 ("Add support to follow HTTP
redirects") to "redirectLimitExceeded".

Also make sure we always use the TransportException(URIish, ...)
constructor; it'll prefix the message given with the sanitized URI.
Change messages to remove the explicit mention of that URI inside the
message. Adapt tests that check the expected exception message text.

For the info logging of redirects, remove a potentially present
password component in the URI to avoid leaking it into the log.

Change-Id: I517112404757a9a947e92aaace743c6541dce6aa
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-23 12:20:55 +02:00
Thomas Wolf 7ac1bfc834 Do authentication re-tries on HTTP POST
There is at least one git server out there (GOGS) that does
not require authentication on the initial GET for
info/refs?service=git-receive-pack but that _does_ require
authentication for the subsequent POST to actually do the push.

This occurs on GOGS with public repositories; for private
repositories it wants authentication up front.

Handle this behavior by adding 401 handling to our POST request.
Note that this is suboptimal; we'll re-send the push data at
least twice if an authentication failure on POST occurs. It
would be much better if the server required authentication
up-front in the GET request.

Added authentication unit tests (using BASIC auth) to the
SmartClientSmartServerTest:

- clone with authentication
- clone with authentication but lacking CredentialsProvider
- clone with authentication and wrong password
- clone with authentication after redirect
- clone with authentication only on POST, but not on GET

Also tested manually in the wild using repositories at try.gogs.io.
That server offers only BASIC auth, so the other paths
(DIGEST, NEGOTIATE, fall back from DIGEST to BASIC) are untested
and I have no way to test them.

* public repository: GET unauthenticated, POST authenticated
  Also tested after clearing the credentials and then entering a
  wrong password: correctly asks three times during the HTTP
  POST for user name and password, then gives up.
* private repository: authentication already on GET; then gets
  applied correctly initially to the POST request, which succeeds.

Also fix the authentication to use the credentials for the redirected
URI if redirects had occurred. We must not present the credentials
for the original URI in that case. Consider a malicious redirect A->B:
this would allow server B to harvest the user credentials for server
A. The unit test for authentication after a redirect also tests for
this.

Bug: 513043
Change-Id: I97ee5058569efa1545a6c6f6edfd2b357c40592a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-22 23:57:09 +02:00
Shawn Pearce 44a75d9ea8 reftable: explicitly store update_index per ref
Add an update_index to every reference in a reftable, storing the
exact transaction that last modified the reference.  This is necessary
to fix some merge race conditions.

Consider updates at T1, T3 are present in two reftables.  Compacting
these will create a table with range [T1,T3].  If T2 arrives during
or after the compaction its impossible for readers to know how to
merge the [T1,T3] table with the T2 table.

With an explicit update_index per reference, MergedReftable is able to
individually sort each reference, merging individual entries at T3
from [T1,T3] ahead of identically named entries appearing in T2.

Change-Id: Ie4065d4176a5a0207dcab9696ae05d086e042140
2017-08-21 15:39:08 -07:00
Shawn Pearce 2d76df2442 reftable: reserve standard PackExt
Reserve "ref" extension for reftable files.  This allows them to be
used in a DFS repository as a stream in a DfsPackDescription.

Change-Id: Ife781bb64d0bb063333183ad2be70a41a2482513
2017-08-17 15:06:51 -07:00
Shawn Pearce 0aae64ce74 reftable: resolve symbolic references
resolve(Ref) helps callers recursively chase symbolic references and
is a useful function when wrapping a Reftable inside a RefDatabase, as
RefCursor does not resolve symbolic references during iteration.

Change-Id: I1ba143f403773497972e225dc92c35ecb989e154
2017-08-17 15:06:51 -07:00
Shawn Pearce 195541dd30 reftable: support threshold based compaction
Transactions may wish to merge several tables together as part of an
operation.  Setting a byte limit allows the transaction to consider
only some recent tables, bounding the cost of the compaction.

Change-Id: If037f2cbdc174ff1a215d5917178b33cde4ddaba
2017-08-17 15:06:51 -07:00
Shawn Pearce d48ac5bf01 reftable: compact merged tables
A compaction of reftables is just copying the results of a
MergedReftable into a ReftableWriter.  Wrap this up into a utility.

Change-Id: I6f5677d923e9628993a2d8b4b007a9b8662c9045
2017-08-17 15:06:51 -07:00
Shawn Pearce 77d8eead6d reftable: merge-join reftables
MergedReftable combines multiple reference tables together in a stack,
allowing higher/later tables to shadow earlier/lower tables.  This
forms the basis of a transaction system, where each transaction writes
a new reftable containing only the modified references, and readers
perform a merge on the fly to get the latest value.

Change-Id: Ic2cb750141e8c61a8b2726b2eb95195acb6ddc83
2017-08-17 15:06:51 -07:00
Shawn Pearce 0398f3dd6e reftable: debug tools
Simple debug programs to experiment with the reftable file format:

  debug-read-reftable
  debug-write-reftable
  debug-verify-reftable
  debug-benchmark-reftable

Change-Id: I79db351d86900f1e58b17e922e195dff06ee71f1
2017-08-17 15:06:51 -07:00
Shawn Pearce 0a26dcf4a3 reftable: scan and lookup reftable files
ReftableReader provides sequential scanning support over all
references, a range of references within a subtree (such as
"refs/heads/"), and lookup of a single reference.  Reads can be
accelerated by an index block, if it was created by the writer.

The BlockSource interface provides an abstraction to read from the
reftable's backing storage, supporting a future commit to connect
to JGit DFS and the DfsBlockCache.

Change-Id: Ib0dc5fa937d0c735f2a9ff4439d55c457fea7aa8
2017-08-17 15:06:51 -07:00
Shawn Pearce 0ecc8367e6 reftable: create and write reftable files
This is a simple writer to create reftable formatted files.  Follow-up
commits will add support for reading from reftable, debugging
utilities, and tests.

Change-Id: I3d520c3515c580144490b0b45433ea175a3e6e11
2017-08-17 15:06:50 -07:00
Thomas Wolf e17bfc96f2 Add support to follow HTTP redirects
git-core follows HTTP redirects so JGit should also provide this.

Implement config setting http.followRedirects with possible values
"false" (= never), "true" (= always), and "initial" (only on GET, but
not on POST).[1]

We must do our own redirect handling and cannot rely on the support
that the underlying real connection may offer. At least the JDK's
HttpURLConnection has two features that get in the way:

* it does not allow cross-protocol redirects and thus fails on
  http->https redirects (for instance, on Github).
* it translates a redirect after a POST to a GET unless the system
  property "http.strictPostRedirect" is set to true. We don't want
  to manipulate that system setting nor require it.

Additionally, git has its own rules about what redirects it accepts;[2]
for instance, it does not allow a redirect that adds query arguments.

We handle response codes 301, 302, 303, and 307 as per RFC 2616.[3]
On POST we do not handle 303, and we follow redirects only if
http.followRedirects == true.

Redirects are followed only a certain number of times. There are two
ways to control that limit:

* by default, the limit is given by the http.maxRedirects system
  property that is also used by the JDK. If the system property is
  not set, the default is 5. (This is much lower than the JDK default
  of 20, but I don't see the value of following so many redirects.)
* this can be overwritten by a http.maxRedirects git config setting.

The JGit http.* git config settings are currently all global; JGit has
no support yet for URI-specific settings "http.<pattern>.name". Adding
support for that is well beyond the scope of this change.

Like git-core, we log every redirect attempt (LOG.info) so that users
may know about the redirection having occurred.

Extends the test framework to configure an AppServer with HTTPS support
so that we can test cloning via HTTPS and redirections involving HTTPS.

[1] https://git-scm.com/docs/git-config
[2] 6628eb41db
[3] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

CQ: 13987
Bug: 465167
Change-Id: I86518cb76842f7d326b51f8715e3bbf8ada89859
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-17 22:16:44 +02:00
Christian Halstrick be767fd7d9 Merge "Fix off-by-one error in Strings.count()" 2017-08-16 06:24:43 -04:00
Christian Halstrick c71af0c73a Merge "Use relative paths for attribute rule matching" 2017-08-16 06:24:33 -04:00
Thomas Wolf b13a285098 Send a detailed event on working tree modifications
Currently there is no way to determine the precise changes done
to the working tree by a JGit command. Only the CheckoutCommand
actually provides access to the lists of modified, deleted, and
to-be-deleted files, but those lists may be inaccurate (since they
are determined up-front before the working tree is modified) if
the actual checkout then fails halfway through. Moreover, other
JGit commands that modify the working tree do not offer any way to
figure out which files were changed.

This poses problems for EGit, which may need to refresh parts of the
Eclipse workspace when JGit has done java.io file operations.

Provide the foundations for better file change tracking: the working
tree is modified exclusively in DirCacheCheckout. Make it emit a new
type of RepositoryEvent that lists all files that were modified or
deleted, even if the checkout failed halfway through. We update the
'updated' and 'removed' lists determined up-front in case of file
system problems to reflect the actual state of changes made.

EGit thus can register a listener for these events and then knows
exactly which parts of the Eclipse workspace may need to be refreshed.

Two commands manage checking out individual DirCacheEntries themselves:
checkout specific paths, and applying a stash with untracked files.
Make those two also emit such a new WorkingTreeModifiedEvent.

Furthermore, merges may modify files, and clean, rm, and stash create
may delete files.

CQ: 13969
Bug: 500106
Change-Id: I7a100aee315791fa1201f43bbad61fbae60b35cb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-15 16:52:00 -04:00
Matthias Sohn 81d020aba9 Merge branch 'stable-4.8'
* stable-4.8:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: Ica20f5aa0d8a365fe3317765b93520b3abd5d342
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-15 00:48:44 +02:00
Matthias Sohn 758a181b82 Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: I1e4fcf84506ff4316567bbb1713e84d8d196c2a1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-15 00:24:49 +02:00
Matthias Sohn 53becf1f59 Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: I669fe48ce0034f9ea1977d38ee39099497422c1c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 23:50:52 +02:00
Matthias Sohn 985e3c6414 Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
  Fix exception handling for opening bitmap index files

Change-Id: Ifb511238e3e98b1bc9f79a990807b940a17ebaa6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 23:43:05 +02:00
Christian Halstrick 1ed1e40387 Fix exception handling for opening bitmap index files
When creating a new PackFile instance it is specified whether this pack
has an associated bitmap index file or not. This information is cached
and the public method getBitmapIndex() will always assume a bitmap index
file must exist if the cached data tells so. But it may happen that the
packfiles are repacked during a gc in a different process causing the
packfile, bitmap-index and index file to be deleted. Since JGit still
has an open FileHandle on the packfile this file is not really deleted
and can still be accessed. But index and bitmap index file are deleted.
Fix getBitmapIndex() to invalidate the cached packfile instance if such
a situation occurs.

This problem showed up when a gerrit server was serving repositories
which where garbage collected with native git regularly. Fetch and
clone commands for certain repositories failed permanently after a
native git gc had deleted old bitmap index files.

Change-Id: I8e620bec74dd3f310ba42024f9a657062f868f0e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 21:09:48 +02:00
Thomas Wolf 37908321c0 Do not apply pushInsteadOf to existing pushUris
Per the git config documentation[1], pushInsteadOf is ignored when
a remote has explicit pushUris.

Implement this, and adapt tests.

Up to now JGit mistakenly applied pushInsteadOf also to existing
pushUris. If some repositories had relied on this mis-feature,
pushes may newly suddenly fail (the uncritical case; the config
just needs to be fixed) or even still succeed but push to unexpected
places, namely to the non-rewritten pushUrls (the critical case).

The release notes should point out this change.

[1] https://git-scm.com/docs/git-config

Bug: 393170
Change-Id: I38c83204d2ac74f88f3d22d0550bf5ff7ee86daf
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 17:27:05 +02:00
Thomas Wolf df3469f6ad Record submodule paths with untracked changes as FileMode.GITLINK
Bug: 520702
Change-Id: I9bb48af9e8f1f2ce7968a82297c7c16f1237f987
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 14:03:51 +02:00
Thomas Wolf f5a2c77dc4 Fix handling of pushInsteadOf
According to [1], pushInsteadOf is

1. applied to the uris, not to the pushUris
2. ignored if a remote has an explicit pushUri

JGit applied it only to the pushUris. As a result, pushInsteadOf was
ignored for remotes having only a uri, but no pushUri.

This commit implements (1) if there are no pushUris. I did not dare
implement (2) because:

* there are explicit tests for it that expect that pushInsteadOf gets
  applied to existing pushUrls, and
* people may actually use and rely on this JGit behavior.

[1] https://git-scm.com/docs/git-config

Bug: 393170
Change-Id: I6dacbf1768a105190c2a8c5272e7880c1c9c943a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 05:40:47 -04:00
Christian Halstrick 196915dde5 Merge "Ensure EOL stream type is DIRECT when -text attribute is present" 2017-08-14 03:34:57 -04:00
Thomas Wolf b07db60908 Fix off-by-one error in Strings.count()
Change-Id: I0667b1624827d1cf0cc1b81f86c7bb44eafd68a7
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 08:04:56 +02:00
Shawn Pearce 53dd9a9e4b Rename extensions.refsStorage to refStorage
This matches the proposal that has been discussed at length on
git-core mailing list and seems to be the accepted convention.

Change-Id: I9f6ab15144826893d1e2a4b48a2d657d6dd445ec
2017-08-11 18:20:50 -07:00
Thomas Wolf a489a8ae9a Ensure EOL stream type is DIRECT when -text attribute is present
Otherwise fancy combinations of attributes (binary or -text in
combination with crlf or eol) may result in the corruption of binary
data.

Bug: 520910
Change-Id: I3ffc666c13d1b9d2ed987b69a67bfc7f42ccdbfc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-11 22:56:50 +02:00
Thomas Wolf 4bc539a814 Use relative paths for attribute rule matching
Attribute rules must match against the entry path relative to the
attribute node containing the rule. The global entry path is to be
used only for the init and the global node (and of course the root
node).

Bug: 520677
Change-Id: I80389a2dc272a72312729ccd5358d7c75e1ea20a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-11 21:59:49 +02:00
Shawn Pearce ed29dec1ea Expose LongMap in util package
This is a useful primitive collection type like IntList.

Change-Id: I04b9b2ba25247df056eb3a1725602f1be6d3b440
2017-08-09 10:42:09 -07:00
Shawn Pearce 40c9c59e07 NB: encode and decode 24-bit ints
Change-Id: Ie036dc46e5a88a4e87dc52e880505bbe34601ca7
2017-08-09 10:42:09 -07:00
Shawn Pearce 22201e8cca Update thread-safety warning about Repository
Change-Id: I1026a77cc688467d5a89a41121146f1bd3d56fa5
2017-08-08 06:44:35 -07:00
Dave Borowitz 8bbe34f27c ReflogWriter: Minor cleanup
Remove unnecessary finals, use consistent punctuation in Javadoc, reflow
some lines, etc.

Change-Id: Ic64db41c86917725ac649022290621406156bcc4
2017-08-02 16:52:34 -04:00
Dave Borowitz cf9662cdfe Eliminate SectionParser construction boilerplate
Happily, most anonymous SectionParser implementations can be replaced
with FooConfig::new, as long as the constructor takes a single Config
arg. Many of these, the non-public ones, can in turn be inlined. A few
remaining SectionParsers can be lambdas.

Change-Id: I3f563e752dfd2007dd3a48d6d313d20e2685943a
2017-08-02 16:50:57 -04:00
Matthias Sohn 3eaa8d8e2a Silence API errors caused by adding enum constants in dbb137e
Change-Id: I46a29eae7b617f3f43f270c40072a1c103ef77f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-01 23:26:42 +02:00
David Pursehouse 4085646f6d Merge changes I424295df,Ib003f7c8
* changes:
  Treat RawText of binary data as file with one single line.
  Trim boilerplate in RawParseUtils_LineMapTest.
2017-08-01 10:18:48 -04:00
Han-Wen Nienhuys a551b64694 Treat RawText of binary data as file with one single line.
This avoids executing mergeAlgorithm.merge on binary data, which is
unlikely to be useful.

Arguably, binary data should not make it to
ResolveMerger#contentMerge, but this approach has the following
advantages:

* binary detection is exact, since it doesn't only look at the start
  of the blob.

* it is cheap, as we have to iterate over the bytes anyway to find
  '\n'.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I424295df1dc60a719859d9d7c599067891b15792
2017-08-01 16:00:46 +02:00
Terry Parker 8c6a9a286e Merge "Use w1 for hashCode of AbbreviatedObjectId" 2017-07-28 19:24:11 -04:00
David Pursehouse 8391cc233b Merge "IntList: support contains(int)" 2017-07-28 14:18:21 -04:00
David Pursehouse 9f462a9914 Merge "Replace findbugs by spotbugs" 2017-07-28 13:47:21 -04:00
Shawn Pearce 4a00f18e8e Use w1 for hashCode of AbbreviatedObjectId
Very short abbreviations that are under 8 hex digits do not
have values in w2. Use w1 as the Java hashCode() instead, so
that the prefix of the abbreviation is always included in the
hashing function used by any java.util.Collection type.

Change-Id: Idaf69f86b62630ba4a022d31b4c293c6d138f557
2017-07-28 10:20:45 -07:00
Shawn Pearce 652a6b0334 IntList: support contains(int)
LongList supports contains(long).
IntList should also support contains(int).

Change-Id: Ic7a81c3c25b0f10d92087b56e9f200b676060f63
2017-07-28 10:18:27 -07:00
Matthias Sohn de7698476b Replace findbugs by spotbugs
SpotBugs [1] is the spiritual successor of FindBugs, carrying on from
the point where it left off with support of its community.

[1] http://spotbugs.readthedocs.io/

Change-Id: I127f2c54b04265b6565e780116617ffa8a4d7eaf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-07-28 16:15:54 +01:00
Dave Borowitz 45da0fc6f7 RefDirectory: Add in-process fair lock for atomic updates
In a server scenario such as Gerrit Code Review, there may be many
atomic BatchRefUpdates contending for locks on both the packed-refs file
and some subset of loose refs. We already retry lock acquisition to
improve this situation slightly, but we can do better by using an
in-process lock. This way, instead of retrying and potentially exceeding
their timeout, different threads sharing the same Repository instance
can wait on a fair lock without having to touch the disk lock. Since a
server is probably already using RepositoryCache anyway, there is a high
likelihood of reusing the Repository instance.

Change-Id: If5dd1dc58f0ce62f26131fd5965a0e21a80e8bd3
2017-07-28 11:03:32 -04:00
Dave Borowitz 6f23210781 RefDirectory: Retry acquiring ref locks with backoff
If a repo frequently uses PackedBatchRefUpdates, there is likely to be
contention on the packed-refs file, so it's not appropriate to fail
immediately the first time we fail to acquire a lock. Add some logic to
RefDirectory to support general retrying of lock acquisition.

Currently, there is a hard-coded wait starting at 100ms and backing off
exponentially to 1600ms, for about 3s of total wait. This is no worse
than the hard-coded backoff that JGit does elsewhere, e.g. in
FileUtils#delete. One can imagine a scheme that uses per-repository
configuration of backoff, and the current interface would support this
without changing any callers.

Change-Id: I4764e11270d9336882483eb698f67a78a401c251
2017-07-28 07:53:25 -04:00
David Pursehouse 5188c23104 Merge "Fix committing empty commits" 2017-07-28 06:08:33 -04:00
David Pursehouse 94aebcb949 Merge "Support overriding a batch's reflog on a per-ReceiveCommand basis" 2017-07-28 06:07:08 -04:00
Christian Halstrick da0770fdec Fix committing empty commits
Allow to explicitly create an empty commit even if committing only
certain files.

Bug: 510685 
Change-Id: If9bf664d7cd824f8e5bd6765fa6cc739af3d7721
2017-07-28 10:46:42 +01:00
David Pursehouse 7e4946626e Merge changes from topic 'batch-ref-update-reflog'
* changes:
  BatchRefUpdate: Expand javadocs and add @Nullable
  PackedBatchRefUpdate: Write reflogs
  Extract constants for reflog entry message prefixes
2017-07-28 05:40:45 -04:00
Zhen Chen b0695e5b7b Add commit check for head references
Make sure all refs/heads/* point to a commit object.

Change-Id: I9c7cf347aaf63d5ef604d520c2383c6cf3043890
Signed-off-by: Zhen Chen <czhen@google.com>
2017-07-26 10:12:37 -07:00
Zhen Chen 673acfc6bd Add connectivity check from references
Make sure all objects referenced by references are reachable. Stop at
the first missing object.

Change-Id: Ifcd7392c4321b17d9290bd87f038bc62bc10dabb
Signed-off-by: Zhen Chen <czhen@google.com>
2017-07-26 10:12:37 -07:00
Zhen Chen 2c2999643f Add dfs fsck implementation
JGit already had some fsck-like classes like ObjectChecker which can
check for an individual object.

The read-only FsckPackParser which will parse all objects within a pack
file and check it with ObjectChecker. It will also check the pack index
file against the object information from the pack parser.

Change-Id: Ifd8e0d28eb68ff0b8edd2b51b2fa3a50a544c855
Signed-off-by: Zhen Chen <czhen@google.com>
2017-07-26 10:12:29 -07:00
Dave Borowitz 104107bf43 Support overriding a batch's reflog on a per-ReceiveCommand basis
Change-Id: I86a4b8f6b4f85b2bae64c1b121e4ee527d46de83
2017-07-26 11:40:15 -04:00
Dave Borowitz a1e11461cc BatchRefUpdate: Expand javadocs and add @Nullable
Change-Id: I22d739a9677e24f36323dceadf7d375ac2f446e8
2017-07-26 11:39:39 -04:00
Dave Borowitz 22e9106224 PackedBatchRefUpdate: Write reflogs
On-disk reflogs are not stored in the packed-refs file, so we cannot
ensure atomic updates. We choose the lesser evil of dropping failed
reflog updates on the floor, rather than throwing an exception even
though the underlying ref updates succeeded.

Add tests for reflogs to BatchRefUpdateTest.

Change-Id: Ia456ba9e36af8e01fde81b19af46a72378e614cd
2017-07-26 11:39:33 -04:00
Dave Borowitz dbb137e0f3 Extract constants for reflog entry message prefixes
Document explicitly that these are untranslated to (mostly) match C git.

Change-Id: I3abcffb4fd611d053bf4373e5d6a14a66f7b9b6b
2017-07-25 13:14:50 -04:00
Dave Borowitz 26962861d4 Implement atomic BatchRefUpdates for RefDirectory
The existing packed-refs file provides a mechanism for implementing
atomic multi-ref updates without any changes to the on-disk format or
lockfile protocol. We just need to make sure that there are no loose
refs involved in the transaction, which we can achieve by packing the
refs while holding locks on all loose refs. Full details of the
algorithm are in the PackedBatchRefUpdate javadoc.

This change does not implement reflog support, which will come in a
later change.

Change-Id: I09829544a0d4e8dbb141d28c748c3b96ef66fee1
2017-07-25 13:14:50 -04:00
Dave Borowitz cf9e3fad52 Separate RefUpdate.Result.REJECTED_{MISSING_OBJECT,OTHER_REASON}
ReceiveCommand.Result has a slightly richer set of possibilities, so it
makes sense for RefUpdate.Result to have more values in order to match.
In particular, this allows us to return REJECTED_MISSING_OBJECT from
RefUpdate when an object is missing.

The comment in RefUpdate#safeParse about expecting some old objects to be
missing is only applicable to the old ID, not the new ID. A missing new
ID is a bug or programmer error, and we should not update a ref to point
to one.

Fix various tests that started failing because they depended for no good
reason on setting refs to point to nonexistent objects; it's always easy
to create a real object when necessary.

It is possible that some downstream users of RefUpdate.Result might
choose to handle one of the new statuses differently, for example by
providing a more user-readable error message; that is not done in this
change.

Change-Id: I734b1c32d5404752447d9e20329471436ffe05fc
2017-07-25 13:12:34 -04:00
David Pursehouse 4940ea14b7 Add missing newlines at ends of Java files
Change-Id: Iead36f53d57ead0eb3edd3f9efb63b6630c9c20c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-07-25 10:37:21 +01:00
Joan Goyeau 826e22e7cc Fix JGit set core.fileMode to false by default instead of true for non Windows OS.
Bug: 519887
Change-Id: I4ae0d6783a9dc62f78ead54ddd1ab2b5b66a811c
Signed-off-by: Joan Goyeau <joan@goyeau.com>
2017-07-24 13:57:21 +01:00
Dmitry Pavlenko 843e444561 Fix matching ignores and attributes pattern of form a/b/**.
Fix patch matching for patterns of form a/b/** : this should not match
paths like a/b but still match a/b/ and a/b/c.

Change-Id: Iacbf496a43f01312e7d9052f29c3f9c33807c85d
Signed-off-by: Dmitry Pavlenko <pavlenko@tmatesoft.com>
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-07-24 09:16:33 +01:00
David Pursehouse ba91e8a086 Merge changes from topic 'packed-batch-ref-update'
* changes:
  Add tests for updating single refs to missing objects
  Fix deleting symrefs
  RefDirectory: Throw exception if CAS of packed ref list fails
  ReceiveCommand: Explicitly check constructor preconditions
  BatchRefUpdate: Document when getPushOptions is null
2017-07-24 03:38:42 -04:00
Shawn Pearce ad269ae426 Merge "Make 'inCoreLimit' of LocalFile used in ResolveMerger configurable" 2017-07-22 18:45:56 -04:00
Changcheng Xiao 1baf86d4d2 Make 'inCoreLimit' of LocalFile used in ResolveMerger configurable
This change makes it possible to configure the 'inCoreLimit' of LocalFile
used in ResolveMerger#insertMergeResult. Since LocalFile itself has some
risks, e.g. it may be left behind as garbage in case of failure. It should
be good to be able to control the size limit for using LocalFile.

Change-Id: I3dc545ade370b2bbdb7c610ed45d5dd4d39b9e8e
Signed-off-by: Changcheng Xiao <xchangcheng@google.com>
2017-07-22 21:51:12 +02:00