Commit Graph

5349 Commits

Author SHA1 Message Date
David Pursehouse d6aec5da31 ObjectCheckerTest: Factor duplicate instances out to constants
The tests:

- testCheckBlobNotCorrupt
- testCheckBlobCorrupt

create instances of ObjectChecker that are the same.

The tests:

- testCheckBlobWithBlobObjectCheckerNotCorrupt
- testCheckBlobWithBlobObjectCheckerCorrupt

also create instances of ObjectChecker that are the same.

Factor these instances out to constants instead of creating them
in the tests.

The `checker` member is still created anew in each test, since some
of the tests change its state.

Change-Id: I2d90263829d01d208632185b1ec2f678ae1a3f4c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-06 00:02:28 +02:00
David Pursehouse 17e849145c SubmoduleConfigTest: Add additional tests in fetchRecurseToConfigValue
Add tests for "true" and "false" matching to "YES" and "NO".

Change-Id: I58223855022871ac4b21bd34ff6a9cd00fce30a1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-05 23:30:18 +02:00
David Pursehouse f9ac86fde2 SubmoduleConfig: Simplify #toConfigValue
Just return configValue instead of transforming name().

Change-Id: I7f94ab2e206e93d1370467e187c5e68e8f6a8836
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-05 23:30:17 +02:00
Shawn Pearce d26309c4d9 Merge changes from topic 'dfs-reftable'
* changes:
  dfs: write reftable from DfsGarbageCollector
  dfs: compact reftables during DfsPackCompactor
2017-09-05 12:39:53 -04:00
Shawn Pearce d13dfac9dc dfs: write reftable from DfsGarbageCollector
If a ReftableConfig has been supplied by the caller, write out a
reftable as a sibling of the the GC pack, alongside the heads.

To bootstrap from a non-reftable system, the refs are read from the
DfsRefDatabase if no GC reftables are present.  Its assumed the
references are fully current, and do not need to be merged with any
other reftables.  Any non-GC reftables will be pruned at the end of
the GC cycle, just like any packs that were replaced.

If a GC reftable is present, all existing reftables are compacted, and
references from DfsRefDatabase are only used to seed the packer.  Its
assumed these are consistent with each other.

Change-Id: Ie397eb58aaaefb6865c816d9b39de3ac12998019
2017-09-05 09:10:16 -07:00
Shawn Pearce d126bcc5c8 dfs: compact reftables during DfsPackCompactor
Combine intermediate, non-GC reftables when combining pack files.
This shrinks the reftable stack, improving lookup times.

Change-Id: I5dbba41806f99af5ecaff3a3119f6630e9404256
2017-09-05 09:05:08 -07:00
Matthias Sohn a0558b7094 Remove final modifier on args4j option or argument fields
Remove final field modifier since args4j does no longer allow use of
final fields [1].

[1] 6e11f89d40

Change-Id: Ib3dee10828d87d6b558c2e90387eadd9aa2ce260
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-09-05 00:06:38 +02:00
Thomas Wolf 11c476346d Fix Daemon.stop() to actually stop the listener thread
ServerSocket.accept() is not interruptible: a thread busy in accept()
may not react to Thread.interrupt() and may not return from accept()
via an InterruptedException. Close the socket instead to make the
daemon's listener thread terminate.

* Close the listening socket to get the listening thread to exit
  instead of interrupting it.
* Add a stopAndWait() method that stops the listening thread and
  then waits until it has indeed finished.
* Set SO_REUSE_ADDRESS on the listening socket.

Bug: 376369
Change-Id: I9d6014103e6dcb0173daea134feb44dc52c5c69a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-04 23:50:48 +02:00
Shawn Pearce de4e0acc30 Merge "Remove workaround for bug in Java's ReferenceQueue" 2017-09-04 14:35:58 -04:00
Matthias Sohn 39b193b6f4 Remove workaround for bug in Java's ReferenceQueue
Sun's Java 5, 6, 7 implementation had a bug [1] where a Reference can be
enqueued and dequeued twice on the same reference queue due to a race
condition within ReferenceQueue.enqueue(Reference).

This bug was fixed for Java 8 [2] hence remove the workaround.

[1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6837858
[2] http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/858c75eb83b5

Change-Id: I2deeb607e3d237f9f825a207533acdee305c7e73
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-09-04 14:44:12 +02:00
Thomas Wolf 6d27869659 Use Config.getEnum() to read the IgnoreSubmoduleMode
Doing so goes through the TypedConfigGetter and thus allows library
clients (for instance EGit) to warn about invalid configurations.

Change-Id: If1080ad90b8aff54a903d4d75637614faad6469b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-04 09:19:03 +02:00
Thomas Wolf 06ea633c18 Don't assume name = path in .gitmodules
While parsing .gitmodules, the name of the submodule subsection is
purely arbitrary: it frequently is the path of the submodule, but
there's no requirement for it to be. By building a map of paths to
the section name in .gitmodules, we can more accurately return
the submodule URL.

Bug: 508801
Change-Id: I8399ccada1834d4cc5d023344b97dcf8d5869b16
Also-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-04 09:19:03 +02:00
David Pursehouse 7317432c31 Upgrade Maven compiler plugins
- error_prone_core to 2.1.1
- maven-compiler-plugin to 3.6.2
- plexus-compiler-javac{-error-prone} to 2.8.2

Change-Id: I20d864c869f2d03f623ea054b6d08e7ee0eb262a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-02 12:11:35 +09:00
David Pursehouse 4e913fcee6 Upgrade args4j to 2.33 in the bazel build
Change-Id: Ic440657e3da53b643f79fbcdde45d4e659bc166d
2017-09-01 12:51:37 +09:00
Matthias Sohn 0b75d0f0f1 Add org.apache.commons.codec 1.9.0 to target platform
This is needed to run tests in org.eclipse.jgit.http.test from Eclipse.
The change 7ac1bfc8 which added this dependency to
org.eclipse.jgit.http.test was already merged.

Restrict dependency to org.apache.commons.codec to the
version range [1.6.0,2.0.0).

CQ: 14048
Change-Id: I461a5f6bfc114757061d68992f9bc7ab38622328
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-09-01 11:03:18 +09:00
David Pursehouse 8abb47b296 Merge "Fix some tests for running in bazel" 2017-08-31 22:02:21 -04:00
David Pursehouse 37ea57701b Merge "Update args4j to 2.33" 2017-08-31 22:01:04 -04:00
David Pursehouse 53deba1b4f FsckPackParser: Fix warning about int being boxed to Integer
Change-Id: I525164090e3990ce353bbb1627aaafaad2c903b6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-01 00:14:32 +09:00
Thomas Wolf c91c20f36f Fix some tests for running in bazel
Some tests call out to external cgit. Those tests all failed for me
locally on Mac. Turned out that the reason was that the system git
config used by the git in the bazel run contained paths with ~/ but
somehow $HOME was not set. As a result the external git returned
with exit code 128.

Fix this by passing along $HOME explicitly. Also improve assertions
to make sure we do get the stderr of the external command in the
test log.

I hadn't noticed that until now because apparently the maven build
does pass along $HOME.

Change-Id: I7069676d5cc7b23a71e79a4866fe8acab5a405f4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-31 15:48:10 +02:00
David Pursehouse de5eb5e2cd FsckPackParser: Fix "Empty block should be documented" warnings
Add "Do nothing" comments, consistent with other empty methods in
the same class.

Change-Id: I27a13a402e94104af617be0e14d8982e75fa73bd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
David Pursehouse 9c1fcf6ad7 FsckPackParser: Fix warnings about variable hiding
Change-Id: Id1ada1cedfad5671e79df303c6dbcb6017b777ce
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
David Pursehouse f8c2067410 Silence API errors caused by adding enum constants in 1baf86d4d2
Change-Id: I23338c04567f95aedd909ab310e01277e8f7db69
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
Thomas Wolf c506f8d2dd Add dependency to jsch for OpenSshConfigTest to bazel build
Make jsch visible to the test bundle and add the dependency.

Change-Id: I0c49ee9b8f64fe8a8c74d2f08865917eb33069b4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-31 14:14:05 +02:00
Shawn Pearce f4329b09d4 Merge changes from topic 'dfs-reftable'
* changes:
  dfs: helper to open multiple reftables
  dfs: expose DfsReftable from DfsObjDatabase
  dfs: support reading reftables through DfsBlockCache
2017-08-30 20:42:23 -04:00
Shawn Pearce 0ca1fb0fe0 Merge changes from topic 'fsck'
* changes:
  DfsFsck: reduce memory usage during verifyIndex
  DfsFsck: refactor pack verify into its own method
  DfsFsck: run connectivity check pass exactly once
2017-08-30 08:42:37 -04:00
Shawn Pearce e5db7c1f0e DfsFsck: reduce memory usage during verifyIndex
Don't convert a lot of ObjectId to String stored in generic
java.util.HashSet.  This is a very expensive way to store objects.

Instead rely on "this" from the FsckPackParser to lookup information
about the objects in this pack file, which lets the verify code avoid
sorting the object list.

Use ObjectIdOwnerMap, which is the most efficient format JGit has
for storing lots of objects.

Change-Id: Ib68f93acb4d91b96d0a44c0612f704500d332ac1
2017-08-30 03:11:53 -07:00
Shawn Pearce 69588c21fe DfsFsck: refactor pack verify into its own method
This simplifies the logic about allocation of the DfsReader, and
clarifies the code considerably by using smaller scopes with less
indentation.

A few static imports from PackExt and slightly shorter variable names
make for a more understandable-at-glance implementation.

Change-Id: Iaf5a0e14fe0349215d9e44446f68d1129ad3bb3d
2017-08-30 03:11:43 -07:00
Shawn Pearce 153c11a49b DfsFsck: run connectivity check pass exactly once
The simpler algorithm is to load all branch tips into an ObjectWalk
and run that walk exactly once.  This avoids redoing work related to
parsing and considering trees reused across side branches.

Move the connectivity check into its own helper method.  This moves it
left one level of identation, and makes it easier to fit the method's
logic with less line wrapping.

Add a "Counting objects..." progress monitor around this phase.  Its
what is used when a server receives a push and is also trying to
verify the client sent all required objects.

Change-Id: I4d53d75d0cdd1a13fff7d513a6ae0b2d14ea4090
2017-08-29 18:14:51 -07:00
Matthias Sohn 3af57b52a3 Update args4j to 2.33
CQ: 11068
Change-Id: I7d52cddacff05477f646fda5f7b9f2de844922f6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-30 03:07:35 +02:00
Robin Stocker c02c9e8042 Fix compilation errors with args4j 2.0.23 and later
The multiValued attribute on @Option was removed. When the field is a
List, it's not actually needed (even with earlier versions of args4j),
see RmTest. In other cases, we have a custom handler, where it's also
not needed.

Bug: 413163
Change-Id: I4bb951e9fab5f4ae4271bd7e11be799dc234ab80
2017-08-30 03:07:35 +02:00
Matthias Sohn f4c5a54aa9 Cleanup: Remove redundant type arguments
Change-Id: I81ce2e25666f5acfce4d3ada6b71ae8c81a97a4d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:34 +02:00
Matthias Sohn e271e2f688 Cleanup: Remove unnecessary $NON-NLS$ tags
Change-Id: I143c9d62b89322509123d4e06a221dd4dc158c55
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:34 +02:00
Matthias Sohn e9fb111182 Cleanup: Organize imports
Change-Id: I6065e59519bc42bd18f5cc5ee3ec5210764ab03c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:33 +02:00
Matthias Sohn 41baa7a791 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Do not automatically organize imports using a save action since this
seems to be buggy and removed some annotations org.eclipse.jgit.pgm
needs to use args4j.

Change-Id: I5a91292c3b9241ce2dde3e4ecce14ad460097129
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:18 +02:00
Matthias Sohn 1052678372 Cleanup: Remove trailing whitespace
Change-Id: I7c08b60fb6afafd1874210320da3da7519632e08
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 01:24:48 +02:00
Matthias Sohn 9906f09868 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Revert the following save actions which were introduced in c0ad77d8:
- always use braces around blocks
- remove unused imports

Other than I expected save actions are run globally on edited files -
and not only on edited code lines only.

Hence revert the save action "Convert control statement bodies to
blocks" which would affect a large number of code lines not affected by
the change editing some small part of a class. This would generate a
large number of changes which may lead to many unnecessary conflicts.
Total number of affected lines across jgit would be around 10k lines.

Also revert "Remove unused imports" since it erroneously removes imports
of some annotations needed by pgm classes using args4j.

Change-Id: I879a47f68e664129e6124cf25c1ae1f6a2d7a5aa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 01:24:47 +02:00
Thomas Wolf dbef8e2537 Pass along the original exception when an ssh connection fails
Otherwise, the stack trace doesn't really tell anything.
See for instance [1].

[1] https://www.eclipse.org/forums/index.php/t/1088535/

Change-Id: If22f2c63c36fec6b32818d2c2acecf20531b4185
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-29 15:08:08 -04:00
Shawn Pearce 4f7ce2b46b dfs: helper to open multiple reftables
ReftableStack maintains multiple open reftables
in an AutoCloseable format, making it easier for
higher level code to handle multiple files.

Change-Id: I7ac35c18e67b7e771fb3de29169d1ee50fab62ca
2017-08-28 15:07:05 -07:00
Shawn Pearce 1a7b8a11df dfs: expose DfsReftable from DfsObjDatabase
Reftable storage in DFS is related to pack storage.  Reftables are
stored in the same namespace, but with PackExt.REFTABLE.  Include
the set of DfsReftable instances in the PackList and export some
helpers to access the tables.

Change-Id: I6a4f5f953ed6b0ff80a7780f4c6cbcc5eda0da3e
2017-08-28 15:07:04 -07:00
Shawn Pearce 1222f34506 dfs: support reading reftables through DfsBlockCache
DfsBlockCache directly shares its internal byte[] with ReftableReader,
avoding copying between the DfsBlockCache and the BlockReader
instances used by ReftableReader.

Change-Id: Icaa4f40052b26f952681414653a8b5314b7c2c23
2017-08-28 15:07:04 -07:00
Shawn Pearce d684ade3d3 Merge "reftable: explicitly store update_index per ref" 2017-08-28 17:57:13 -04:00
David Pursehouse 10a8df22fa Merge "Enhance Eclipse save actions" 2017-08-28 12:08:01 -04:00
Terry Parker cb24de07d0 Merge "Add BlobObjectChecker" 2017-08-28 12:00:53 -04:00
Matthias Sohn c0ad77d84c Enhance Eclipse save actions
Add the following Eclipse save actions executed when saving modified
lines. This should help to reduce manual work needed to maintain a clean
and consistent code style:
- organize imports
- always use braces around blocks
- add missing annotations
  - @Override including implementation of interface methods
  - @Deprecated
- remove
  - unused imports
  - unnecessary $NON-NLS$ tags
  - redundant type arguments

Also add default values for new settings that were introduced in recent
Eclipse versions up to Neon since we updated save rules the last time.

Change-Id: Idc90b249df044d0552f04edf01a5f607c4846f50
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-28 11:52:45 -04:00
Masaya Suzuki fd74cf2f78 Add BlobObjectChecker
Some repositories can have a policy that do not accept certain blobs. To
check if the incoming pack file contains such blobs, ObjectChecker can
be used. However, this ObjectChecker is not called by PackParser if the
blob is stored as a whole. This is because the object can be so large
that it doesn't fit in memory.

This change introduces BlobObjectChecker. This interface takes chunks of
a blob instead of the entire object. ObjectChecker can optionally return
a BlobObjectChecker. This won't change existing ObjectChecker
implementation; existing implementation continues to receive deltified
blob objects only.

Change-Id: Ic33a92c2de42bd7a89786a4da26b7a648b25218d
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2017-08-28 08:42:27 -07:00
Thomas Wolf 1637c44048 FetchCommand: pass on CredentialsProvider to submodule fetches
When a JGit API command is implemented in terms of other API
commands, the child command must "inherit" all relevant settings.
Calling configure() ensures that the CredentialsProvider and the
connection timeout are propagated correctly.

Bug: 515325
Change-Id: I948e306693a9edb7b199a735877413b6eddcfba4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:37:43 +02:00
Thomas Wolf d031b64667 Exclude file matching: fix backtracking on match failures after **
** matching always tries the empty match first. If a mismatch occurs
later, the ** must be extended by exactly one segment and matching must
resume with the matcher following the ** matcher.

Bug: 520920
Change-Id: Id019ad1c773bd645ae92e398021952f8e961f45c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:41 +02:00
Thomas Wolf d80b999c76 Fix path pattern matching to work also for gitattributes
Path pattern matching for attribute rules is different than matching
for excluded files.

The first difference concerns patterns without slashes. For
gitattributes those must match on the last component only, not on
any earlier segment. This is true also for directory-only patterns.

The second difference concerns directory-only patterns. Those also
must not match on a prefix or segment except the last one. They do
not apply recursively to all files beneath.

And third, matches only on a prefix must match for gitattributes
only if the last matcher was "/**".

Add a new parameter for such path matching to IMatcher.matches() and
pass it through as appropriate (false for gitignore, true for
gitattributes). As far as gitignore is concerned, there is no change.

New tests have been added, and some existing attribute matching tests
have been fixed since they operated on wrong assumptions.

Bug: 508568
Change-Id: Ie825dc2cac8a85a72a7eeb0abb888f3193d21dd2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:40 +02:00
Thomas Wolf 08d4bef6b7 Add new tests for gitignore/gitattribute pattern matching
These tests verify that JGit matches the same as C git, for
both attribute matching (.gitattributes) and file exclusion matching
(.gitignore). These tests work by setting up a test repository and
test rules, and then determine excluded files or attributes both with
JGit and with the native C git, and then compare the results.

For .gitignore tests, we run

  git ls-files --ignored --exclude-standard -o
  
and for attribute tests we use

  git check-attr --stdin --all
  
and pass the list of all files in the repository via stdin.

Change-Id: I5b40946e04ff4a97456be7dffe09374323b7c89d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:40 +02:00
Thomas Wolf 426caf99ee Ignore invalid TagOpt values
C git silently ignores invalid tagopt values; so make JGit behave the
same way.

Bug: 429625
Change-Id: I99587cc46c7e0c19348bcc63f602038fa9a7f378
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 09:11:03 +02:00