Commit Graph

3236 Commits

Author SHA1 Message Date
Shawn Pearce 398d8e877f RefTree: Change peel suffix to " ^" (space carrot)
Using ^{} as the peel suffix has caused problems when projects used
tags like v2.1 and then v2.1.1, v2.2.2, etc.  The peeled value for
v2.1 was stored very far away in the tree relative to v2.1 itself as
^ sorts in the ASCII/UTF-8 encoding after all other common tag
characters like digits and dots.

Use " ^" instead as space is not valid in a reference name, sorts
before all other valid reference characters (thus forcing next entry
locality) and this looks like a peeled marker for the prior tag.

Change-Id: I26d2247a0428dfe26a9c319c02159502b3a67455
2016-01-11 15:27:14 -08:00
Shawn Pearce 088c2fc6e3 FileRepository: Support extensions.refsBackendType = RefTree
This experimental code can be enabled in $GIT_DIR/config:

  [core]
    repositoryformatversion = 1

  [extensions]
    refsBackendType = RefTree

When these are set the repository will read references from the
RefTree rooted by the $GIT_DIR/refs/txn/committed reference.

Update debug-rebuild-ref-tree to rebuild refs/txn/committed only from
the bootstrap layer.  This avoids misuse by rebuilding using packed-refs
and $GIT_DIR/refs tree.

Change-Id: Icf600e4a36b2f7867822a7ab1f1617d73c710a4b
2016-01-11 15:27:07 -08:00
Shawn Pearce 48e245fc60 RefTreeDatabase: Ref database using refs/txn/committed
Instead of storing references in the local filesystem rely on the
RefTree rooted at refs/txn/committed.  This avoids needing to store
references in the packed-refs file by keeping all data rooted under
a single refs/txn/committed ref.

Performance to scan all references from a well packed RefTree is very
close to reading the packed-refs file from local disk.

Storing a packed RefTree is smaller due to pack file compression,
about 49.39 bytes/ref (on average) compared to packed-refs using
~65.49 bytes/ref.

Change-Id: I75caa631162dc127a780095066195cbacc746d49
2016-01-11 15:26:42 -08:00
Shawn Pearce 8f31aa7c28 Merge "RevCommit: Better support invalid encoding headers" 2016-01-11 15:48:16 -05:00
Shawn Pearce 31d92ace5b RevCommit: Better support invalid encoding headers
With this support we no longer need the 'utf-8' alias. UTF-8 will be
automatically tried when the encoding header is not recognized and used
if the character sequence cleanly decodes as UTF-8.

Modernize some of the references to use StandardCharsets.

Change-Id: I4c0c88750475560e1f2263180c4a98eb8febeca0
2016-01-11 12:30:35 -08:00
Shawn Pearce 0f8743d4d7 Remove deprecated Tree, TreeEntry, FileTreeEntry and friends
These types were deprecated in 0.9.1 (aka 384a19eee0).
If anyone is still using them, its time to stop.

Change-Id: I3f73347ba78c639e0c6a504812bc1a0702f829b1
2016-01-09 09:45:33 -08:00
Shawn Pearce 1243e25aad Paths.pathCompare: Utility to sort paths from byte[]
Consolidate copies of this function into one location.

Add some unit tests to prevent bugs that were accidentally
introduced while trying to make this refactoring.

Change-Id: I82f64bbb8601ca2d8316ca57ae8119df32bb5c08
2016-01-09 09:44:57 -08:00
Shawn Pearce bace383507 Add Paths utility class
Simple container for some path related utility functions.

Change-Id: Ice2bec6ad12b1e2cea15988c01aa9dd4e016a849
2016-01-08 16:58:23 -08:00
Shawn Pearce 73eb32be82 RefTree: Store references in a Git tree
A group of updates can be applied by updating the tree in one step,
writing out a new root tree, and storing its SHA-1. If references
are stored in RefTrees, comparing two repositories is a matter of
checking if two SHA-1s are identical. Without RefTrees comparing two
repositories requires listing all references and comparing the sets.

Track the "refs/" directory as a root tree by storing references
that point directly at an object as a GITLINK entry in the tree.
For example "refs/heads/master" is written as "heads/master".

Annotated tags also store their peeled value with ^{} suffix, using
"tags/v1.0" and "tags/v1.0^{}" GITLINK entries.

Symbolic references are written as SYMLINK entries with the blob of
the symlink carrying the name of the symbolic reference target.

HEAD is outside of "refs/" namespace so it is stored as a special
"..HEAD" entry. This name is chosen because ".." is not valid in
a reference name and it almost looks like "../HEAD" which names
HEAD if the reader was inside of the "refs/" directory.

A new Command type is required to handle symbolic references and
peeled references.

Change-Id: Id47e5d4d32149a9e500854147edd7d93c1041a39
2016-01-07 15:40:26 -08:00
Shawn Pearce 5001038f52 Merge "InMemoryRepository: Abort BatchRefUpdate if a command previously failed" 2016-01-06 15:18:04 -05:00
Shawn Pearce 4c9eda17be InMemoryRepository: Abort BatchRefUpdate if a command previously failed
If any command has already been marked as failing, fail the entire batch.

Change-Id: I1692240841aa4f4cb252bdccbc6f11d9246929c1
2016-01-06 12:05:46 -08:00
Shawn Pearce f0d634eed7 DFS: Allow other RefDatabase implementations
Permit a DfsRepository implementation to use a different RefDatabase
than DfsRefDatabase.

Change-Id: Ia263285f547bde1943993cc994d0222185021a16
2016-01-06 11:54:45 -08:00
Matthias Sohn 91a3f17474 Merge changes Idaed9310,I62bd7c0b
* changes:
  ObjectChecker: honor some git-core fsck.* options
  ObjectChecker: allow some objects to skip errors
2016-01-03 10:27:37 -05:00
Shawn Pearce 163be57d0f Merge changes from topic 'add-df'
* changes:
  DirCache: Do not create duplicate tree entries
  DirCacheEditor: Replace file-with-tree and tree-with-file
  AddCommand: Use NameConflictTreeWalk to identify file-dir changes
2016-01-01 12:58:35 -05:00
Shawn Pearce 09500165a8 Fix "remote: Counting objects: ..." formatting
Trailing whitespace is usually removed in properties files so
JGitText did not supply a space between : and the remote message.
Ensure the space exists at runtime by reading the localized string
and appending a space if it is missing.

Messages should be dynamically fetched and not held in a static
class variable, as they can be changed using thread locals.

Change-Id: If6a3707d64094253b1a5304fbfafcf195db7497a
2015-12-31 17:37:16 -08:00
David Ostrovsky 13502fef8f Implement Buck driven build
Today there are plenty of modern build tool systems available in the
wild (in no particular order):

* http://bazel.io
* https://pantsbuild.github.io
* http://shakebuild.com
* https://ninja-build.org
* https://buckbuild.com

The attributes, that all these build tools have in common, are:

* reliable
* correct
* very fast
* reproducible

It must not always be the other build tool, this project is currently
using. Or, quoting Gerrit Code Review maintainer here:

  "Friends, don't let friends use <the other build tool system>!"

This change is non-complete implementation of JGit build in Buck,
needed by Gerrit Code Review to replace its dependency with standlone
JGit cell. This is very useful when a developer is working on both
projects and is trying to integrate changes made in JGit in Gerrit.

The supported workflow is:

  $ cd jgit
  $ emacs <hack>
  $ cd ../gerrit
  $ buck build --config repositories.jgit=../jgit gerrit

With --config repositories.jgit=../jgit jgit cell is routed through
JGit development tree.

To build jgit, issue:

  $ buck build //:jgit
  [-] PROCESSING BUCK FILES...FINISHED 0,0s

Yes, you can't measure no-op build time, given that Buck daemon is
used.

Change-Id: I301a71b19fba35a5093d8cc64d4ba970c2877a44
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
2015-12-31 10:08:55 -08:00
Shawn Pearce 2fbd98dbe8 Merge "Fix encoding problem from curl repostory on github" 2015-12-30 23:47:28 -05:00
Shawn Pearce 80edcac06f Merge "Fix hanging fetch via SSH" 2015-12-30 23:27:09 -05:00
Eryk Szymanski 53c3bbe42f Fix encoding problem from curl repostory on github
Pushing curl repository to gerrit fails with a message:
remote: error: internal error while processing changes
java.nio.charset.IllegalCharsetNameException: 'utf8'

curl repository url: https://github.com/bagder/curl.git

To avoid this problem encodingAliases in RawParseUtils have
been extended to contain "'utf8'" (single quoted utf8) string.

Change-Id: I40f613cfdcabf0dc9455bee45116ab8d8c7dd6ee
Signed-off-by: Eryk Szymanski <eryksz@gmail.com>
2015-12-30 23:24:12 -05:00
Shawn Pearce ac41920a43 ObjectChecker: honor some git-core fsck.* options
Accept some of the same section keys that fsck does in git-core,
allowing repositories to skip over specific kinds of acceptable
broken objects, e.g.:

  [fsck]
    duplicateEntries = ignore
    zeroPaddedFilemode = ignore

The zeroPaddedFilemode = ignore is a synonym for the JGit specific
allowLeadingZeroFileMode = true. Only accept the JGit key if git-core
key was not specified.

Change-Id: Idaed9310e2a5ce5511670ead1aaea2b30aac903c
2015-12-30 15:19:09 -08:00
Shawn Pearce fa7ce0e0f3 ObjectChecker: allow some objects to skip errors
Some ancient objects may be broken, but in a relatively harmless way.
Allow the ObjectChecker caller to whitelist specific objects that are
going to fail checks, but that have been reviewed by a human and decided
the objects are OK enough to permit continued use of.

This avoids needing to rewrite history to scrub the broken objects out.

Honor the git-core fsck.skipList configuration setting when receiving a
push or fetching from a remote repository.

Change-Id: I62bd7c0b0848981f73dd7c752860fd02794233a6
2015-12-30 15:18:48 -08:00
Matthias Sohn 3fc93f8a56 Rename files using NIO2 atomic rename
Bug: 319233
Change-Id: I5137212f5cd3195a52f90ed5e4ce3cf194a13efd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-30 17:22:33 -05:00
Shawn Pearce e3acf01748 ObjectChecker: use java.text.Normalizer directly
Base Java version for JGit is now Java 7. The java.text.Normalizer
class was available in Java 6. Reflection is no longer required to
normalize strings for Mac OS X.

Change-Id: I98e14b72629a7a729a2d40a3aa275932841274e8
2015-12-30 15:45:13 -05:00
Shawn Pearce 7177b506b4 Merge changes I3db332bf,I614e7110,I37a4ccd6,I86b81e7f,Ic8b362e0, ...
* changes:
  Sort "eager" path-like options to the end of the help
  reset command: provide convenient and meaningful options help
  commit command: allow to specify path(s) argument(s)
  status command: consume more then one argument after --
  repo command: properly name the required 'path' argument
  Un-ignored existing CLI tests which run just fine on Java 7+
  Don't treat command termination due '-h' option as a fatal error
2015-12-30 15:44:30 -05:00
Shawn Pearce 018a91b715 Unify fetch and receive ObjectChecker setup
This avoids duplication of code between receive-pack and fetch-pack paths.
Separate methods are still required to check use of receive.fsckobjects vs.
fetch.fsckobjects, both of which default to transfer.fsckobjects.

Change-Id: I41193e093e981a79fc2f63914e273aaa44b82162
2015-12-29 16:55:41 -08:00
Shawn Pearce 29aa444760 PackWriter: use lib.ObjectIdSet to avoid wrapper
Hoist ObjectIdSet up to lib as part of the public API and add
the interface to some common types like PackIndex and JGit custom
ObjectId map types.  This cleans up wrapper code in a number of
places by allowing direct use of the types as an ObjectIdSet.

Future commits can now rely on ObjectIdSet as a simple read-only
type to check a set of objects from a number of storage options.

Change-Id: Ib62b062421d475bd52abd6c84a73916ef36e084b
2015-12-29 15:32:24 -08:00
Shawn Pearce 683c41af92 DirCache: Do not create duplicate tree entries
If a file (e.g.  "A") and a subtree file (e.g. "A/foo.c") both appear
in the DirCache this cache should not be written out as a tree object.
The "A" file and "A" subtree conflict with each other in the same tree
and will fail fsck.

Detect this condition during DirCacheBuilder and DirCacheEditor
finish() so the application can be halted early before it updates a
DirCache that might later write an invalid tree structure.

Change-Id: I95660787e88df336297949b383f4c5fda52e75f5
2015-12-29 11:33:39 -08:00
Shawn Pearce b71ba69410 DirCacheEditor: Replace file-with-tree and tree-with-file
If a PathEdit tries to store a file where a subtree was, or a subtree
where a file was, replace the entry in the DirCache with the new
name(s).  This supports switching between file and tree entry types
using a DirCacheEditor.

Add new unit tests to cover the conditions where these can happen.

Change-Id: Ie843d9388825f9e3d918a5666aa04e47cd6306e7
2015-12-29 11:33:39 -08:00
Shawn Pearce 3776b14ab4 AddCommand: Use NameConflictTreeWalk to identify file-dir changes
Adding a path that already exists but is changing type such as
from symlink to subdirectory requires a NameConflictTreeWalk to
match up the two different entry types that share the same name.

NameConflictTreeWalk needs a bug fix to pop conflicting entries
when PathFilterGroup aborts the walk early so that it does not
allow DirCacheBuilderIterator to copy conflicting entries into
the output cache.

Change-Id: I61b49cbe949ca8b4b98f9eb6dbe7b1f82eabb724
2015-12-29 11:33:35 -08:00
Andrey Loskutov 0505657d6a commit command: allow to specify path(s) argument(s)
This fixes the command below:

jgit commit a -m "added file a"

which currently fails with:

org.eclipse.jgit.api.errors.JGitInternalException: The combination of
arguments --all and --only is not allowed

Bug: 484973
Change-Id: I37a4ccd68101a66520ef99110f7aa0cbdcc8beba
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-12-29 15:00:09 +01:00
Andrey Loskutov 241b50be31 Simplify development of commands: added main() to CLIGitCommand
This will execute git commands (with arguments) specified on the command
line, handy for developing/debugging a sequence of arbitrary git
commands working on same repository.

The git working dir path can be specified via Java system property
"git_work_tree". If not specified, current directory will be used.

Change-Id: I621a9ec198c31e28a383818efeb4b3f835ba1d6f
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-12-28 22:59:59 +01:00
Shawn Pearce ef757a7e12 DirCacheEditor: Cleanup DeleteTree constructor
Neaten up formatting and avoid strings, which prevents the need for
NLS comment tags.  Instead check the last character using char
literal, and append a char literal instead of a string.

Change-Id: Ib68e017769a1f5c03200354a805769d585a48c8b
2015-12-28 11:43:31 -08:00
Shawn Pearce 110d3ca595 DirCacheEditor: Fix formatting to avoid , at start of line
Change-Id: I1b1d614470c67fe4736fdc9c26ae26fb38dd58b5
2015-12-28 11:43:30 -08:00
Shawn Pearce 21cec184b7 Merge "AddCommand: Avoid unnecessary string conversions" 2015-12-28 14:24:33 -05:00
Christian Halstrick 2fdce1ef8c Merge "AddCommand: Cleanup conditional logic" 2015-12-28 08:39:07 -05:00
Shawn Pearce a94e517940 AddCommand: Avoid unnecessary string conversions
Change-Id: I13634caeccd9f675a86adfdfa94099b6fb75463a
2015-12-23 22:26:50 -08:00
Shawn Pearce bb8627bd9b AddCommand: Cleanup conditional logic
Unnest and simplify conditional logic for handling entries.

Change-Id: I3093cab5f0edfaf3efbbd6c644e9c922edc67d38
2015-12-23 20:31:15 -08:00
Dmitry Neverov 9c71bf14b7 Close copy threads in case of errors
Bug: 484775
Change-Id: I3c7105188e615b6b994261f4ece0c8abc98eb444
Signed-off-by: Dmitry Neverov <dmitry.neverov@gmail.com>
2015-12-24 01:20:14 +01:00
Yuxuan 'fishy' Wang d88695e412 Skip nested copyfiles in RepoCommand.
Similar to nested directories, nested copyfiles won't work with git submodule
either.

Change-Id: Idbe965ec20a682fca0432802858162f8238f05de
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>
2015-12-22 14:05:35 -08:00
Andrey Loskutov 1aafa61958 Checkout should be able to override modified symbolic links
Handle existing symlink as a file, not as directory if deleting a file
before creating (overriding) a symlink.

Bug: 484491
Change-Id: I29dbf57d1daec2ba98454975b093e1d381d05196
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-12-17 10:42:44 +01:00
Shawn Pearce 8cfbbc9f54 Merge "ProgressSpinner: Simple busy wait entertainment" 2015-12-16 16:20:59 -05:00
Shawn Pearce a38531b21c ProgressSpinner: Simple busy wait entertainment
Keep a user amused while the server does work by spinning a
little ASCII-art object on a single line.

Change-Id: Ie8f181d1aa606d4ae69e5d3ca4db387cea739f38
2015-12-16 15:43:07 -05:00
James Kolb 3096a13340 Included cached deltas in delta packStatistics.
Previously, non-reuse deltas were only included in packStatistics if they
were not cached by the deltaWindow.

Change-Id: I7684d8214875f0a7569b34614f8a3ba341dbde9c
Signed-off-by: James Kolb <jkolb@google.com>
2015-12-16 14:06:16 -05:00
Jonathan Nieder cfa0b2fb3c Merge changes Ib4d53bdd,I55bd512c
* changes:
  Do not let PathFilter.create("a/b") match 'a' unless 'a' is a subtree
  Add tests for PathFilterGroup.Single
2015-12-16 12:09:49 -05:00
Shawn Pearce a3e9876d8f Merge "Fix InterruptTimer leak in BasePackConnection" 2015-12-16 10:29:02 -05:00
Matthias Sohn 6340c76d8b Fix InterruptTimer leak in BasePackConnection
When setting timeout on push, BasePackConnection creates a timer, which
will be terminated when push finishes. But, when using
SmartHttpPushConnection, it dropped the first timer created in the
constructor and then created another timer in doPush. If new threads are
created faster than the gc collects then this may stop the service if
it's hitting the max process limit. Hence don't create a new timer if it
already exists.

Bug: 474947
Change-Id: I6746ffe4584ad919369afd5bdbba66fe736be314
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-16 14:19:38 +01:00
Shawn Pearce 7c5b2761ed Merge "Refspec: loosen restrictions on wildcard "*"" 2015-12-15 23:22:12 -05:00
Shawn Pearce 9eda23e469 Merge "BaseRepositoryBuilder should trim CR from .git symref" 2015-12-15 23:18:14 -05:00
Shawn Pearce 8b749f7de5 Merge "Fix possible arithmetic overflow when setting a timeout" 2015-12-15 23:16:21 -05:00
Shawn Pearce fcd796a9a0 Merge "Accept UTF8 BOM with BlobBasedConfig" 2015-12-15 23:12:56 -05:00
Shawn Pearce 6297c2e92c Merge "Bug 484342: Support @ in username in SSH url." 2015-12-15 23:09:31 -05:00
Jonathan Nieder 75a0dcac18 Do not let PathFilter.create("a/b") match 'a' unless 'a' is a subtree
PathFilter and PathFilterGroup form JGit's implementation of git's
path-limiting feature in commands like log and diff.  To save time
when traversing trees, a path specification

	foo/bar/baz

tells the tree walker not to traverse unrelated trees like qux/.  It
does that by returning false from include when the tree walker is
visiting qux and true when it is visiting foo.

Unfortunately that test was implemented to be slightly over-eager: it
doesn't only return true when asked whether to visit a subtree "foo"
but when asked about a plain file "foo" as well.  As a result, diffs
and logs restricted to some-file/non-existing-suffix unexpectedly
match against some-file:

 $ jgit log -- LICENSE/no-such-file
 commit 629fd0d594
 Author: Shawn O. Pearce <spearce@spearce.org>
 Date:   Fri Jul 02 14:52:49 2010 -0700

     Clean up LICENSE file
[...]

Fix it by checking against the entry's mode.

Gitiles +log has the same bug and benefits from the same fix.

Callers know not to worry about what subtrees are included in the tree
walk because shouldBeRecursive() returns true in this case, so this
behavior change should be safe.  This also better matches the behavior
of C git:

 $ empty=$(git mktree </dev/null)
 $ git diff-tree --abbrev $empty HEAD -- LICENSE/no-such-file
 $ git diff-tree --abbrev $empty HEAD -- tools/no-such-file
 :000000 040000 0000000... b62648d... A  tools

Bug: 484266
Change-Id: Ib4d53bddd8413a9548622c7b25b338d287d8889d
2015-12-15 15:22:10 -08:00
Mike Gilbode 3e03907f65 Bug 484342: Support @ in username in SSH url.
Change-Id: I5795e925afff796488ba26c83694e806b76a374f
Signed-off-by: Mike Gilbode <gilbode@gmail.com>
2015-12-15 10:44:16 -05:00
Matthias Sohn 2647d024af Fix push with jgit pgm failing with "unauthorized"
Pushing with JGit commandline to e.g. Github failed with "unauthorized"
since HttpUrlConnection calls the configured authenticator implicitly.

The problem is that during a push two requests are sent to the server,
first a GET and then a POST (containing the pack data). The first GET
request sent anonymously is rejected with 401 (unauthorized). When an
Authenticator is installed the java.net classes will use the
Authenticator to ask the user for credentials and retry the request.
But this happens under the hood and JGit level code doesn't see that
this happens.

The next request is the POST but since JGit thinks the first GET request
went through anonymously it doesn't add authentication headers to the
POST request. This POST of course also fails with 401 but since this
request contains a lot of body-data streamed from JGit (the pack file!)
the java.net classes can't simply retry the request with authorization
headers. The whole process fails.

Fix this by using Apache httpclient which doesn't use Authenticator to
retrieve credentials. Instead initialize TransportCommand to use the
default credential provider if no other credentials provider was set
explicitly. org.eclipse.jgit.pgm.Main sets this default for the JGit
command line client.

Change-Id: Ic4e0f8b60d4bd6e69d91eae0c7e1b44cdf851b00
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-15 12:49:08 +01:00
Christian Halstrick 5c66ad5684 Merge "NetRCCredentialsProvider should return false if any item is missing" 2015-12-15 06:44:08 -05:00
Christian Halstrick 2f0b675a71 Merge "Fix ChainingCredentialsProvider" 2015-12-15 03:52:09 -05:00
Christian Halstrick 310e858f81 Fix possible arithmetic overflow when setting a timeout
BasePackPushConnection#readStringLongTimeout() was setting a timeout 10
times bigger than some other timeout or the pack transfer time. This
could lead to negative integer values when we hit an arithmetic
overflow. Add a check for this situation and set the timeout to
Integer.MAX_VALUE when overflow happens.

Bug: 484352
CC: Eugene Petrenko <eugene.petrenko@gmail.com>
Change-Id: Ie2a86312c1bcb1ec3e6388fa490ab3c845d41808
2015-12-15 09:44:09 +01:00
Andrey Loskutov 95b36b397b Null-annotated Ref class and fixed related compiler errors
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.

Change-Id: I24c8a600ec962d61d5f40abf73eac4203e115240
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-12-15 09:07:05 +01:00
Matthias Sohn d2044aed4f JGit v4.2.0.201512141825-rc1
Change-Id: Id725ea888acd23793d37de2864ec0727160a2eaf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-15 00:22:13 +01:00
Shawn Pearce 8179185d11 Merge "push: Do not blindly overwrite peer" 2015-12-14 17:45:57 -05:00
Shawn Pearce 4f8c993c49 push: Do not blindly overwrite peer
If an application uses PushConnection directly on the native Git wire
protocols JGit should send along the application's expected oldId, not
the advertised value.  This allows the remote peer to compare-and-swap
since it was not tested inside JGit.

Discovered when I tried to use a PushConnection (bypassing the
standard PushProcess) and the client blindly overwrote the remote
reference, even though my app had supplied the wrong ObjectId for
the expectedOldObjectId. This was not expected and cost me over an
hour of debugging, plus "corruption" in the remote repository.

By passing along the exact expectedOldObjectId from the app the
remote side can do the check that the application skipped, and
avoid data loss.

Change-Id: Id3920837e6c47100376225bb4dd61fa3e88c64db
2015-12-14 09:18:27 -08:00
Christian Halstrick 5b55498b16 Fix FileTreeIterator.idSubmodule(Entry)
FileTreeIterator was calling by mistake
WorkingTreeIterator.idSubmodule(Entry). Instead it should always compute
idSubmodule on its own.

Change-Id: Id1b988aded06939b1d7edd2671e34bf756896c0e
2015-12-13 01:24:46 +01:00
Andrey Loskutov 85d09a9ec7 URIish: fixed full uri pattern not expecting end of line after host name
Bug: 483326
Change-Id: I8b6e3eb648c8ec2c38f73de22382537b1276b779
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-12-12 00:55:31 +01:00
Kaloyan Raev e23521f0d9 Add remote command to JGit CLI
Supported subcommands are:
 - <none> (lists available remotes)
 - add
 - remove
 - set-url
 - update
 
Supported options are:
 --verbose
 --push
 --prune

Bug: 481316
Change-Id: I57c34ed6daabb7d308bc383b17c1ef4af433e714
Signed-off-by: Kaloyan Raev <kaloyan.r@zend.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-11 22:54:48 +01:00
Doug Kelly 618b0c1ceb Accept UTF8 BOM with BlobBasedConfig
In I1f5dc07182dbf6bba2a9f4807fdd25b475da4ead, FileBasedConfig got
support for reading a configuration with UTF8 BOM.  Apply the same
support to BlobBasedConfig, to make SubmoduleWalk able to parse
.gitmodules configurations with BOM.

Change-Id: I25b5474779952fe2c076180b96fc2869eef190a8
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
2015-12-09 16:48:20 -06:00
Matthias Sohn afd167a1f2 NetRCCredentialsProvider should return false if any item is missing
Change-Id: I894d1621aaccd71dfe100fe83a1bd9d50a1e0808
2015-12-08 14:12:26 +01:00
Matthias Sohn 5786cc3c0f Fix ChainingCredentialsProvider
The ChainingCredentialsProvider gave up chaining to the next provider if
the first one returned no credentials items for the given URI.

Change-Id: I9539c50db35e564db9d43d8ebb71d7e9c6fdcc19
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-08 14:12:26 +01:00
Christian Halstrick 6dfad98d72 Merge "Fix NPE in HttpAuthMethod" 2015-12-08 02:25:35 -05:00
Jonathan Nieder e12482deed Use runtime retention for Nullable annotation
JGit's Nullable type was added[1] in the hope of being able to add
nullness annotations that (a) do not preclude building and running
with Java 7 and (b) could be shared by Gerrit, which uses a custom
Nullable type for other reasons[2].  Sharing a type is useful because
Eclipse's null analysis is only able to use one Nullable type at a
time in a given workspace (so for this analysis to function in a
workspace used to develop Gerrit, JGit and Gerrit would need to use
the same Nullable type).

The new Nullable type has CLASS instead of RUNTIME retention because
there wasn't any obvious use for the annotation at run time.

Gerrit uses the Nullable annotation to communicate with Guice.  Guice
injection happens at runtime, so it needs to be able to read the
@Nullable annotations at run time[3].  Otherwise Guice produces
provisioning errors, such as

 3) null returned by binding at com.google.gerrit.lucene.LuceneChangeIndex$Factory.create()
  but parameter 7 of com.google.gerrit.lucene.LuceneChangeIndex.<init>() is not @Nullable

Switch to RUNTIME retention to avoid this.

While at it, update the javadoc to explain more clearly how this
annotation relates to other Nullable types[4].  This should make it
clearer why JGit needed another Nullable type:

 A. Avoiding dependency on Java 8
 B. RUNTIME retention to allow Guice to read the annotation at run time
 C. Named Nullable so Guice can recognize the annotation
 D. Not an addition to Java EE's javax.annotation package, to avoid
    the split-package problem[2] that prevents the annotation from
    being readable at run time when loaded from an OSGi container
 E. Avoiding heavyweight dependencies, deprecated dependencies, and
    dependencies on package internals

 org.checkerframework.checker.nullness.qual.Nullable: A
 com.sun.istack.internal.Nullable: B, E
 *.CheckForNull, *.NullAllowed, etc: C
 edu.umd.cs.findbugs.annotations.Nullable: B, E
 javax.annotation.Nullable: D
 org.eclipse.jdt.annotation.Nullable: B
 org.jetbrains.annotations.Nullable: B
 org.jmlspecs.annotation.Nullable: E
 android.annotation.Nullable, android.support.annotation.Nullable: E

[1] https://git.eclipse.org/r/59993
[2] https://gerrit-review.googlesource.com/50112
[3] https://github.com/google/guice/blob/master/core/src/com/google/inject/internal/Nullability.java
[4] https://github.com/typetools/checker-framework/blob/5832a01f1/checker/src/org/checkerframework/checker/nullness/NullnessAnnotatedTypeFactory.java#L118
    http://types.cs.washington.edu/checker-framework/current/checker-framework-manual.html#nullness-related-work

Change-Id: I6c482653d2b53e3509abb11211b67fc29cf2949c
Signed-off-by: Jonathan Nieder <jrn@google.com>
2015-12-07 19:26:23 -05:00
Matthias Sohn ac89b47eeb Fix NPE in HttpAuthMethod
If the password char array is null constructing a new String from this
array fails with a NPE. Add a null check to fix this.

Change-Id: Ifae6eecca38d5f114861f44658a32521e6e96866
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-07 16:05:19 +01:00
Shawn Pearce 3d8e6b1e16 Support atomic push in JGit client
This should mirror the behavior of `git push --atomic` where the
client asks the server to apply all-or-nothing. Some JGit servers
already support this based on a custom DFS backend. InMemoryRepository
is extended to support atomic push for unit testing purposes.

Local disk server side support inside of JGit is a more complex animal
due to the excessive amount of file locking required to protect every
reference as a loose reference.

Change-Id: I15083fbe48447678e034afeffb4639572a32f50c
2015-12-02 22:06:06 -08:00
Matthias Sohn f109af47d0 Fix wrong @since tag In ReceiveCommand
Change-Id: I58fa657dd4783fed0ffca94020c87c49d99009c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-12-02 15:24:26 +01:00
Shawn Pearce f89f30ffcd ReceiveCommand.filter: Accept Iterable
PreReceiveHook is given a Collection<ReceiveCommand> and it can be
very useful here to call ReceiveCommand.filter(cmds, NOT_ATTEMPTED).

Overload filter to accept both Iterable and List.
Keep backwards binary compatibility for List by upcasting to Iterable.

Change-Id: Ib1341876c703670945ef209edc8259715ee86c26
2015-12-01 19:59:03 -08:00
Shawn Pearce 8078021e8f Merge changes If1490ca2,Ife950253
* changes:
  DirCacheBuilder: Speed up reading from trees
  Delay locating .gitattributes until requested
2015-11-30 19:03:11 -05:00
Shawn Pearce 2d011cd648 DirCacheBuilder: Speed up reading from trees
Recursively copying a tree into a DirCache is a bottleneck for some
algorithms like the in memory merge code in Gerrit Code Review.  Drop
a layer down in the stack and use CanonicalTreeParser directly as the
addition logic only processes 1 tree at a time and does not need the
merge sorting feature (or overhead) of TreeWalk.

Combined with 761814fe9c ("DirCacheEntry: Speed up creation by
avoiding string cast") tree loading 38,900 entries nearly halves
in running time from 70ms to 36ms on some platforms.

Change-Id: If1490ca25de0679a71cf508f59b486f9cc816165
2015-11-30 15:46:40 -08:00
Shawn Pearce 93eca6dfe1 DirCache: Add helper to read from a tree
Application code sometimes wants to read a DirCache from an ObjectId,
but its confusing how to do this because its buried inside the
DirCacheBuilder.

Use this utility in a few places within JGit that also want to read
a DirCache from a tree's ObjectId.

Change-Id: I578b7e18e58753d154937f4ab835012b09e5adca
2015-11-29 22:05:24 -05:00
Shawn Pearce b0eb744604 Delay locating .gitattributes until requested
Instead of checking every entry for .gitattributes only look for the
entry on request by TreeWalk.  This avoids impacting uses like RevWalk
filtering history.

When the attrs is requested skip to the start of the tree and look for
.gitattributes until either it is found, or it is impossible to be
present.  Due to the sorting rules of tree entries .gitattributes
should be among the first or second entries in the tree so very few
entries will need to be considered.

Waiting to find the .gitattributes file by native ordering may miss
attrs for files like .config, which sorts before .gitattributes.
Starting from the front of the tree on demand ensures the attributes
are parsed as early as necessary to process any entry in the tree.

Due to TreeWalk recursively processing up the tree of iterators we
cannot just reset the current CanonicalTreeParser to the start as
parent parsers share the same path buffer as their children.
Resetting a parent to look for .gitattributes may overwrite path
buffer data used by a child iterator.  Work around this by building a
new temporary CanonicalTreeParser instance.

Change-Id: Ife950253b687be325340d27e9915c9a40df2641c
2015-11-29 18:56:49 -08:00
Shawn Pearce 45cc76524b DirCache: Fix bad code formatting
Line breaking before , is ugly to read.  Most formatters and humans
expect line break after , so update a few offensive locations.

Use String.format() for the construction of the error message when
a bad DirCachEntry is being failed on. This simplifies the code and
its not a performance critical section.

Change-Id: I5d990389e7ba24ef0861cf8ec0026ed030d4aeda
2015-11-28 09:28:09 -08:00
Shawn Pearce 761814fe9c DirCacheEntry: Speed up creation by avoiding string cast
The checkPath function is available as a byte[] form, in fact the
String form just converts to byte[] to run the algorithm.

Having DirCacheEntry take a byte[] -> String -> byte[] to check if
each path is valid is a huge waste of CPU time. On some systems it
can double the time required to read 38,999 files from trees to the
DirCache. This slows down any operation using a DirCache.

Expose the byte[] form and use it for DirCacheEntry creation.

Change-Id: I6db7bc793ece99ff3c356338d793c07c061aeac7
2015-11-28 09:01:03 -08:00
Shawn Pearce 885879ffe9 DirCache: Fix getEntriesWithin("") to not include null entries
The internal array may be longer than entryCnt, in this case the tail
of the array is padded with null entries. Do not return those to the
caller of getEntriesWithin().

Change-Id: I19efb05e103fab6b739ced407f6e28155a48dba6
2015-11-27 23:25:22 -08:00
Shawn Pearce 46e4992e92 Fix performance regression in CanonicalTreeParser
Change-Id: I14046559fddb9656d890d3099010117e84cd9439
2015-11-27 23:23:49 -08:00
Christian Halstrick bd31555ee5 Add support for smudge filters
If defined in .gitattributes call smudge filter during checkout.

To support checkout where current HEAD,index do not contain attributes
we need to also consider attributes from the tree we checkout. Therefore
CanonicalTreeParser has to learn how to provide attributes.

Change-Id: I168fdb81a8e1a9f991587b3e95a36550ea845f0a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-27 23:33:53 +01:00
Christian Halstrick 5d9f595eb8 Add support for clean filters
When filters are defined for certain paths in gitattributes make
sure that clean filters are processed when adding new content to the
object database.

Change-Id: Iffd72914cec5b434ba4d0de232e285b7492db868
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-27 23:23:09 +01:00
Ivan Motsch 75697adc5a Add the new class Attributes holding multiple Attribute(s)
Attributes represents a semantic collector of Attribute(s) and replaces
the anonymous Map<String,Attribute>. This class will be returned by
TreeWalk.getAttributes(). It offers convenient access to the attributes
wrapped in the Attributes object. Adds preparations for a future
Attribute Macro Expansion

Change-Id: I8348c8c457a2a7f1f0c48050e10399b0fa1cdbe1
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
2015-11-27 11:40:34 +01:00
Matthias Sohn 2cdc130dfc Add missing @since tags and missing javadoc
Change-Id: I8575797127fc96abea8af56f019ca39f5897486f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-25 19:25:53 -05:00
Christian Halstrick 55fb18feaa Fix FS.runProcess() to close the InputStream
When FS.runProcess was called and an InputStream was given the method
tried to pump the whole InputStream to the process. When the method
ended the InputStream was not giving any data anymore. Consequently
close the InputStream inside the method.

Change-Id: I0ed738a775e5c977b21447d195acee1ecf5e2cb9
2015-11-26 00:36:09 +01:00
Jonathan Nieder fe98218e0d repo: Do not use search path to find refs/remotes/origin/<branch>
When running from a non-bare repository, "jgit repo" checks whether
the rev passed in is a sha1 or branch name and in the latter case will
check out origin/<branch>.

We are expecting refs/remotes/origin/<branch>, but as a side effect of
using getRef we also end up looking for refs/origin/<branch>,
refs/heads/origin/<branch>, and so on.  Avoid that by using exactRef
instead.

Signed-off-by: Jonathan Nieder <jrn@google.com>
Change-Id: I670b2e48a88138a1f2104ea201baa958e9edbddb
2015-11-25 14:05:24 -08:00
Jonathan Nieder 5be4814e38 Repository: Introduce exactRef and findRef, deprecate getRef
The Repository class provides only one method to look up a ref by
name, getRef.  If I request refs/heads/master and that ref does not
exist, getRef will look further in the search path:

 ref/refs/heads/master
 refs/heads/refs/heads/master
 refs/remotes/refs/heads/master

This behavior is counterintuitive, needlessly inexpensive, and usually
not what the caller expects.

Allow callers to specify whether to use the search path by providing
two separate methods:

- exactRef, which looks up a ref when its exact name is known
- findRef, which looks for a ref along the search path

For backward compatibility, keep getRef as a deprecated synonym for
findRef.

This change introduces findRef and exactRef but does not update
callers outside tests to use them yet.

Change-Id: I35375d942baeb3ded15520388f8ebb9c0cc86f8c
Signed-off-by: Jonathan Nieder <jrn@google.com>
2015-11-25 14:05:20 -08:00
Andrey Loskutov 830117e761 Null-annotated RefDatabase class
No other code changes except adding nullness annotations.

Change-Id: If2606fb208f6690bd4fd7ad953e709a3ebd6398c
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-11-25 21:19:05 +01:00
Andrey Loskutov 1020f40813 Null-annotated Repository class and fixed related compiler errors
org.eclipse.jgit.lib.Repository class is an example of the API which
should be written with Java 8 java.util.Optional<T> type. Unfortunately
this API is already released and widely used. The good clients are
currently doing their best with checking return values for null and bad
clients do not know how bad their code is.

I've tried not to change any logic and to be as less intrusive as
possible. Most of the JGit code was well prepared to this, only few
classes needed some smaller fixes.

This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.

Because annotating getDirectory() and getFS() as Nullable would cause
lot of additional changes in JGit and EGit they are postponed.

Change-Id: Ie8369d2c9c5fac5ce83b3b1b9bc217d7b55502a3
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-11-25 20:52:19 +01:00
Matthias Sohn d2faec27a7 Raise error if FileNotFoundException is caught for an existing file
File, FileInputStream and friends may throw FileNotFoundException even
if the file is existing e.g. when file permissions don't allow to access
the file content. In most cases this is a severe error we should not
suppress hence rethrow the FileNotFoundException in this case.

This may also fix bug 451508.

Bug: 451508
Change-Id: If4a94217fb5b7cfd4c04d881902f3e86193c7008
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-24 15:54:20 -05:00
Jonathan Nieder 15e60b646e Merge "RefDirectory.getRef: Treat fake missing symrefs like real ones" 2015-11-24 15:43:06 -05:00
Christian Halstrick 18af2d4265 Let FS_Win32_Cygwin detect symlink support by creating temporary symlink
The class FS_Win32 was always trying out to create a temporary symlink
in order to find out whether symlinks are supported. FS_Win32_Cygwin was
overwriting this method and always returned true. But when the user
running JGit does not have administrative rights then the creation of
symlinks is forbidden even if he is running on FS_Win32_Cygwin. A lot of
tests failed only on the Windows platform because of this. It was
correctly detected that FS_Win32_Cygwin is the filesystem abstraction to
be used but creation of symlinks always failed because of lacking
privileges of the user running the tests.

This fix teaches FS_Win32_Cygwin to behave like FS_Win32 and to test
whether symlinks can be created in order to find out whether symlinks
are supported.

Change-Id: Ie2394631ffc4c489bd37c3ec142ed44bbfcac726
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-21 01:49:49 +01:00
Matthias Sohn a6bcc988e0 Remove no longer needed outdated API warning filter
This fixes a warning saying this filter isn't needed anymore.

Change-Id: If77056378befe86c1773950dbe48a82c833fd532
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-20 18:19:35 -05:00
Jonathan Nieder 1e47c7058d RefDirectory.getRef: Treat fake missing symrefs like real ones
getRef() loops over its search path to find a ref:

	Ref ref = null;
	for (String prefix : SEARCH_PATH) {
		ref = readRef(prefix + needle, packed);
		if (ref != null) {
			ref = resolve(ref, 0, null, null, packed);
			break;
		}
	}
	fireRefsChanged();
	return ref;

If readRef returns null (indicating that the ref does not exist), the
loop continues so we can find the ref later in the search path.  And
resolve should never return null, so if we return null it should mean
we exhausted the entire search path and didn't find the ref.

... except that resolve can return null: it does so when it has
followed too many symrefs and concluded that there is a symref loop:

	if (MAX_SYMBOLIC_REF_DEPTH <= depth)
		return null; // claim it doesn't exist

Continue the loop instead of returning null immediately.  This makes
the behavior more consistent.

Arguably getRef should throw an exception when a symref loop is
detected.  That would be a more invasive change, so if it's a good
idea it will have to wait for another patch.

Change-Id: Icb1c7fafd4f1e34c9b43538e27ab5bbc17ad9eef
Signed-off-by: Jonathan Nieder <jrn@google.com>
2015-11-19 14:52:56 -08:00
Christian Halstrick 4b114d3c6b Throw IndexReadException if existing index can't be read
If the index file exists but can't be read for example because of wrong
filesystem permissions we should throw a specific exception. This allows
EGit to handle this error situation.

Bug: 482607
Change-Id: I50bfcb719c45caac3cb5550a8b16307c2ea9def4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-19 23:05:22 +01:00
Matthias Sohn d3e61db455 Fix pre-push hook to not set null remoteName as first argument
According to [1] the pre-push hook expects two parameters which provide
the name and location of the destination remote, if a named remote is
not being used both values should be the same.

We did set the first parameter to null in that case which caused
ProcessBuilder to throw a NullPointerException since its start() method
doesn't accept null arguments.

[1] https://git-scm.com/docs/githooks#_pre_push

Bug: 482393
Change-Id: Idb9b0a48cefac01abfcfdf00f6d173f8fa1d9a7b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-18 09:50:51 +01:00
Christian Halstrick 69cd6f5864 Introduce FS.execute() to execute a command defined by a ProcessBuilder
Change-Id: I2ad2c71ad30b969455bdea89637b8e996b1dad8c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-11-18 00:28:45 +01:00
Christian Halstrick 6389948a81 Add an attribute accessor to CanonicalTreeParser and use it in Treewalk
When checking out a branch we need to access the attributes stored
in the tree to be checked out. E.g. directly after a clone we checkout
the remote HEAD. In this case index and workingtree are still empty.
So we have to search the tree to be checked out for attributes.

Change-Id: I6d96f5d095ed2e3c259d4b12124e404f5215bd9f
2015-11-18 00:28:45 +01:00
Arthur Daussy 12280c02db Adds the git attributes computation on the treewalk
Adds the getAttributes feature to the tree walk. The computation of
attributes needs to be done by the TreeWalk since it needs both a
WorkingTreeIterator and a DirCacheIterator.

Bug: 342372
CQ: 9120
Change-Id: I5e33257fd8c9895869a128bad3fd1e720409d361
Signed-off-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
2015-11-18 00:28:45 +01:00