Commit Graph

4600 Commits

Author SHA1 Message Date
David Pursehouse 1ce24d46a6 Remove references to org.eclipse.jgit.java7
The bundle org.eclipse.jgit.java7 was removed in 4.0.

Remove references to it from the README.md.

Remove reference to it from org.eclipse.jgit.test/.project, which
causes an error message when opening the project in Eclipse:

  Resource '/org.eclipse.jgit.java7' does not exist.

Change-Id: If0dbd562dcd60550bec3c0f793289474b7624bce
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-08-05 11:22:27 +09:00
Jonathan Nieder 958684edef Merge changes from topic 'shallowClone'
* changes:
  RevWalk: Make fields available to DepthWalk
  Shallow fetch: avoid sending unneeded blobs
2016-08-04 20:26:53 -04:00
Terry Parker 2a7897bb4c RevWalk: Make fields available to DepthWalk
DepthWalk needs to override toObjectWalkWithSameObjects() and thus
needs to be able to directly set the objects and freeFlags fields, so
make them package private.

Change-Id: I24561b82c54ba3d6522582ca25105b204d777074
Signed-off-by: Terry Parker <tparker@google.com>
2016-08-04 17:20:27 -07:00
Terry Parker 7edf05530d Shallow fetch: avoid sending unneeded blobs
When doing an incremental fetch from JGit, "have" commits are marked
as "uninteresting". In a non-shallow fetch, when the RevWalk hits an
"uninteresting" commit it marks the commit's corresponding tree as
uninteresting. That has the effect of dropping those trees and all the
trees and blobs they reference out of the thin pack returned to the
client.

However, shallow fetches use a DepthWalk to limit the RevWalk, which
nearly always causes the RevWalk to terminate before encountering the
"have" commits. As a result the pack created for the incremental fetch
never encounters "uninteresting" tree objects and thus includes
duplicate objects that it knows the client already has.

Change-Id: I7b1f7c3b0d83e04d34cd2fa676f1ad4fec904c05
Signed-off-by: Terry Parker <tparker@google.com>
2016-08-04 17:20:23 -07:00
Terry Parker a8f15b7b2f Merge "PackWriterTest: Improve readability" 2016-08-04 19:19:07 -04:00
Terry Parker 9843489cb8 PackWriterTest: Improve readability
Add wants() and haves() static utility functions to improve readability.


Change-Id: I4d44e17a9af97c0203e2ebe112eabb1f67d272a6
Signed-off-by: Terry Parker <tparker@google.com>
2016-08-04 14:59:05 -07:00
Dave Borowitz d6fe52e914 DiffFormatter: Support setting a reader without a repo
Change-Id: I575cdb9c0a9a341b79ef5e3c7a35e68cde142540
2016-08-03 16:24:37 -04:00
Jonathan Nieder 23b1405484 Merge "RefSpec: Make WildcardMode public" 2016-07-28 14:47:05 -04:00
Stefan Beller 647bf67f8d RefSpec: Make WildcardMode public
We have to be able to access the enum from outside the package as part of
the API.

Change-Id: I4bdc6bd53a14237c5f4fb9397ae850f9a24c4cfb
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-07-28 11:29:38 -07:00
David Pursehouse f703a5eb2c Fix typo in email address in copyright headers
Change-Id: Ib7b73d7d37574682bb58f969822c842e4e579233
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-28 16:05:22 +09:00
David Pursehouse ddb12b003f LfsServerTest: Treat response body as UTF-8 when decoding error message
Change-Id: I495f0b60b7128fff27502641e6a5d05f888d4e8a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-28 12:17:45 +09:00
David Pursehouse bb9988c236 LfsProtocolServlet: Pass request and path to getLargeFileRepository
Passing the request and path to the method will allow implementations
to have more control over determination of the backend, for example:

- return different backends for different requests
- accept or refuse requests based on request characteristics
- etc

Change-Id: I1ec6ec54c91a5f0601b620ed18846eb4a3f46783
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-27 15:54:03 +09:00
David Pursehouse e27eab26e2 FileLfsServlet: Include error message in response body
According to the specification [1], the error response body must
include the error message in json format.

[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md#response-errors

Change-Id: I79e7a841d230fdedefa53b9c6d2d477e81e1f9e6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-27 11:07:05 +09:00
David Pursehouse 2e652aebab FileLfsServlet: Return HTTP 422 instead of 400
According to the specification [1], the error response status code
should be 422 when there is a validation error with one or more of
the objects in the request

[1] https://github.com/github/git-lfs/blob/master/docs/api/v1/http-v1-batch.md#response-errors

Change-Id: Id03fe00a2109b896d9a154228a14a33bce5accc3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-27 11:01:14 +09:00
David Pursehouse b8d861bfd5 Repository: Log negative useCnt message together with stack trace
The message "close() called when useCnt is already zero" is logged with
level warning, and then if debug logging is enabled, the stack trace is
logged separately with level debug.

Log the message and the stack trace in the same call, so that they always
appear together in the output rather than potentially interleaved with
other log statements.

Change-Id: I1b5c1557ddc2d19f3f5b29baec96e62bc467d88a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 18:58:17 +09:00
David Pursehouse c15903ea3e MergeFormatter: Suppress warning about unchecked conversion
The warning can be fixed by adding a type to the argument, but doing so
breaks the API and previous attempts to fix it in that way [1, 2] were
reverted [3, 4].

[1] https://git.eclipse.org/r/#/c/45709/
[2] https://git.eclipse.org/r/#/c/66602/
[3] https://git.eclipse.org/r/#/c/49400/
[4] https://git.eclipse.org/r/#/c/66659/

Change-Id: I01dd52e09da24f70d408ffa96e21c129a79041ea
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 18:58:17 +09:00
David Pursehouse 0d872a0837 Annotate Sets#of with @SafeVarArgs to prevent heap pollution warning
This prevents the warning:

  Potential heap pollution via varargs parameter

The method doesn't do any casting of types that would cause the heap
pollution, so it should be safe to add @SafeVarArgs.

See [1] for information about this warning.

[1] http://stackoverflow.com/a/12462259/381622

Change-Id: Ic6d252915ea44b4f1c385afecb98906cd2c54382
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 18:58:17 +09:00
David Pursehouse eb7a93bb2a Archive: Make project name consistent with other subprojects'
Change-Id: I8341f3340d8129b4f966d541097269210fbf65d2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 18:58:17 +09:00
David Pursehouse 4528d5f56d Ignore 'The value of exception parameter is not used' warning
Change-Id: I50407e4a33e35b718ca40503fdd436f1f9f70fba
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 10:16:49 +09:00
David Pursehouse 053684ac24 Update links to LFS API documentation
The location of the API v1 documentation has changed. Update the
links accordingly.

Change-Id: If0148a0b573c474bbe157fcb7e6674c0055fe8b4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 10:16:49 +09:00
David Pursehouse 0be8021819 Merge branch 'stable-4.4'
* stable-4.4:
  JGit v4.4.1.201607150455-r
  RefDirectory: remove ref lock file for following ref dir removal

Change-Id: Ifc8a782efd7f2f991e70ad2a3691a8dba66c7554
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2016-07-26 10:09:35 +09:00
Stefan Beller a2d3c376a6 RefSpecs: allow construction of weird wildcarded RefSpecs
Gerrit's superproject subscription feature uses RefSpecs to formalize
the ACLs of when the superproject subscription feature is allowed.

As this is a slightly different use case than describing a local/remote
pair of refs, we need to be more permissive. Specifically we want to allow:

    refs/heads/*
    refs/heads/*:refs/heads/master
    refs/heads/master:refs/heads/*

Introduce a new constructor, that allows constructing these RefSpecs.

Change-Id: I46c0bea9d876e61eb2c8d50f404b905792bc72b3
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-07-25 10:10:06 -07:00
Stefan Beller 1556ca740b RefSpec: reject refs ending in '/'
We had a case in Gerrits superproject subscriptions where
'refs/heads/' was configured with the intention to mean 'refs/heads/*'.

The first expression lacks the '*', which is why it is not considered
a wildcard but it was considered valid and so was not found early to be
a typo.

Refs are not allowed to end with '/' anyway, so add a check for that.

Change-Id: I3ffdd9002146382acafb4fbc310a64af4cc1b7a9
Signed-off-by: Stefan Beller <sbeller@google.com>
2016-07-25 10:10:06 -07:00
Terry Parker e9e2f7e6b5 Merge "Push implementation of option strings" 2016-07-22 19:27:20 -04:00
Dan Wang 7f9fb80002 Push implementation of option strings
Example usage:
$ ./jgit push \
  --push-option "Reviewer=j.doe@example.org" \
  --push-option "<arbitrary string>" \
  origin HEAD:refs/for/master
Stefan Beller has also made an equivalent change to CGit:
http://thread.gmane.org/gmane.comp.version-control.git/299872

Change-Id: I6797e50681054dce3bd179e80b731aef5e200d77
Signed-off-by: Dan Wang <dwwang@google.com>
2016-07-22 15:40:54 -07:00
Dave Borowitz ecb2aa0503 DfsObjDatabase: Add lazy last modified method to PackList
Change-Id: Id045f162fa584ea14da29a9df58a42c53a78dc15
2016-07-19 13:18:13 -04:00
Dave Borowitz e790ec3fb1 Merge changes I159e9154,I06c722b2
* changes:
  DfsObjectDatabase: Expose PackList and move markDirty there
  Invalidate DfsObjDatabase pack list when refs are updated
2016-07-19 10:51:52 -04:00
Dave Borowitz 0f1c361e62 DfsObjectDatabase: Expose PackList and move markDirty there
What's invalidated when an object database is "dirty" is not the whole
database, but rather a specific list of packs. If there is a race
between getting the pack list and setting the volatile dirty flag
where the packs are rescanned, we don't need to mark the new pack list
as dirty.

This is a fine point that only really applies if the decision of
whether or not to mark dirty actually requires introspecting the pack
list (say, its timestamps). The general operation of "take whatever
is the current pack list and mark it dirty" may still be inherently
racy, but the cost is not so high.

Change-Id: I159e9154bd8b2d348b4e383627a503e85462dcc6
2016-07-18 15:57:41 -04:00
Dan Wang 0d2a7b7737 BatchRefUpdate: Remove unused namesToCheck variable
This variable has been populated and never used since it was
introduced in commit 5cf53fdacf
(Speed up clone/fetch with large number of refs, 2013-02-18).
Noted by FindBugs:
"BatchRefUpdate.java:359, UC_USELESS_OBJECT, Priority: Normal"

Change-Id: I7aacb49540aaee4a83db3d38b15633bb6c4773d0
Signed-off-by: Dan Wang <dwwang@google.com>
2016-07-15 20:09:43 -04:00
Matthias Sohn df479df176 JGit v4.4.1.201607150455-r
Change-Id: I61dbc29a962c8185fb356fe1ca30a1e673166d47
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-15 10:54:36 +02:00
Christian Halstrick eaedbc3980 Merge "RefDirectory: remove ref lock file for following ref dir removal" into stable-4.4 2016-07-15 04:43:53 -04:00
Matthias Sohn 9a4e8de467 Fix AppServer build errors in Eclipse with <4.6 target platforms
9aa3748 added dummy implementations for loadRoleInfo() and
loadUserInfo() to class MappedLoginService to fix compile errors in
Eclipse when using 4.6 target platform which brings Jetty 9.3 adding
these two methods. Unfortunately this causes errors when using non 4.6
target platform coming with an older Jetty version. Fix this by
extracting the anonymous subclass of MappedLoginService which allows to
suppress the unused private method errors in Eclipse.

Change-Id: I75baeea7ff4502ce9ef2b541b3c0555da5535d79
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-14 22:59:14 +02:00
Dave Borowitz 18e9db306b Invalidate DfsObjDatabase pack list when refs are updated
Currently, there is a race where a user of a DfsRepository in a single
thread may get unexpected MissingObjectExceptions trying to look up an
object that appears as the current value of a ref:

1. Thread A scans packs before scanning refs, for example by reading
   an object by SHA-1.
2. Thread B flushes an object and updates a ref to point to that
   object.
3. Thread A looks up the ref updated in (2). Since it is scanning refs
   for the first time, it sees the new object SHA-1.
4. Thread A tries to read the object it found in (3), using the cached
   pack list it got from (1). The object appears missing.

Allow implementations to work around this by marking the object
database's current pack list as "dirty." A dirty pack list means that
DfsReader will rescan packs and try again if a requested object is
missing. Implementations should mark objects as dirty any time the ref
database reads or scans refs that might be newer than a previously
cached pack list.

Change-Id: I06c722b20c859ed1475628ec6a2f6d3d6d580700
2016-07-14 15:58:14 -04:00
Matthias Sohn ffddf8d437 Merge branch 'stable-4.4'
* stable-4.4:
  Log if Repository.useCnt becomes negative
  Time based eviction strategy for repository cache
  Add method to read time unit from config
  Align include.path max depth with native git
  Config load should not fail on unsupported or nonexistent include path
  Allow using JDK 7 bootclasspath when compiling JGit using Java 8
  Extract work queue to allow reusing it

Change-Id: I6aeedb1cb8b0c3068af344a719c80a03ae68fc23
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-12 17:12:41 +02:00
Matthias Sohn ceaadf8f98 Log if Repository.useCnt becomes negative
We observe in Gerrit 2.12 that useCnt can become negative in rare cases.
Log this to help finding the bug.

Change-Id: Ie91c7f9d190a5d7cf4733d4bf84124d119ca20f7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-12 11:32:50 +02:00
Christian Halstrick 7ffe547da7 Time based eviction strategy for repository cache
When Repository.close() decrements the useCount to 0 currently the cache
immediately evicts the repository from WindowCache and RepositoryCache.
This leads to I/O overhead on busy repositories because pack files and
references are inserted and deleted from the cache frequently.

This commit defers the eviction of a repository from the caches until
last use of the repository is older than time to live. The eviction is
handled by a background task running periodically.

Add two new configuration parameters:
* core.repositoryCacheExpireAfter: cache entries are evicted if the
cache entry wasn't accessed longer than this time in milliseconds
* core.repositoryCacheCleanupDelay: defines the interval in milliseconds
for running a background task evicting expired cache entries. If set to
-1 the delay is set to min(repositoryCacheExpireAfter, 10 minutes). If
set to 0 the time based eviction is switched off and no background task
is started. If time based eviction is switched off the JVM can still
evict cache entries if heap memory is running low.

Change-Id: I4a0214ad8b4a193985dda6a0ade63b70bdb948d7
Also-by: Matthias Sohn <matthias.sohn@sap.com>
Also-by: Hugo Arès <hugo.ares@ericsson.com>
Also-by: Sasa Zivkov <sasa.zivkov@sap.com>
2016-07-12 11:32:49 +02:00
Hugo Arès c1ca9cc800 Add method to read time unit from config
Time units supported:

-milliseconds (1 ms, 2 milliseconds)
-seconds (1 s, 1 sec, 1 second, 2 seconds)
-minutes (1 m, 1 min, 1 minute, 2 minutes)
-hours (1 h, 1 hr, 1 hour, 2 hours)
-days (1 d, 1 day, 2 days)
-weeks (1 w, 1 week, 2 weeks)
-months (1 mon, 1 month, 2 months)
-years (1 y, 1 year, 2 years)

This functionality is implemented in Gerrit ConfigUtil class. Add it to
JGit so it can eventually be remove from Gerrit.

Change-Id: I2d6564ff656b6ab9424a9360624061c94fd5f413
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-12 11:32:49 +02:00
Matthias Sohn 9aa3748e4c Implement new abstract MappedLoginService methods added in Jetty 9.3
Eclipse Neon comes with Jetty 9.3 which is causing unimplemented
abstract method errors in test class AppServer when using the JGit or
EGit Neon target platform. Fix this by adding dummy implementations.

Change-Id: Ie49107d814a846997de95f149e91fe1ec2fbe4d8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-11 00:36:50 +02:00
Jonathan Nieder ca2052a8c1 DfsGarbageCollector: avoid closing idx and bitmap streams twice
These try-with-resources blocks close the underlying output stream
twice: once when closing the CountingOutputStream wrapper, then again
when closing the DfsOutputStream out.

Simplify by only closing the CountingOutputStream.

In practice this shouldn't matter because the close() method of a
Closable is required to be idempotent, but avoiding the redundant
extra close makes the code simpler to read and understand.

Change-Id: I1778c4fc8ba075a2c6cd2129528bb272cb3a1af7
2016-07-07 16:08:02 -07:00
Matthias Sohn 450c0af42f Fix unclosed resource warnings in FileTreeIteratorTest
Change-Id: I75ea7deca64a707cd6b5c61c3c83062f20041684
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-07 17:36:55 -04:00
Matthias Sohn 4c25757df7 Fix assertion in SmartClientSmartServerTest.testPush_CreateBranch()
This assertion only defined a message but didn't assert anything.

Change-Id: I0914642b64b69dc4e3ec24acbf8052f9171613d8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2016-07-07 17:23:33 -04:00
Shawn Pearce 63cf87f863 ReceivePack: report protocol parsing failures on channel 3
If the client sent a well-formed enough request to see it wants to use
side-band-64k for status reporting (meaning its a modern client), but
any other command record was somehow invalid (e.g. corrupt SHA-1)
report the parsing exception using channel 3.  This allows clients to
see the failure and know the server will not be continuing.

git-core and JGit clients send all commands and then start a sideband
demux before sending the pack. By consuming all commands first we get
the client into a state where it can see and respond to the channel 3
server failure.

This behavior is useful on HTTPS connections when the client is buggy
and sent a corrupt command, but still managed to request side-band-64k
in the first line.

Change-Id: If385b91ceb9f024ccae2d1645caf15bc6b206130
2016-07-05 17:22:37 -07:00
Shawn Pearce a1bedf0822 ReceivePack: catch InvalidObjectIdException while parsing shallow
The "shallow $id" parsing can also throw InvalidObjectIdException,
just like parseCommand. Move it into its own method with a proper
try-catch block to convert to the checked PackProtocolException.

Change-Id: I6839b95f0bc4fbf4d2c213331ebb9bd24ab2af65
2016-07-05 17:17:50 -07:00
Shawn Pearce 0373180321 ReceivePack: remove unnecessary try-catch around parseCommand
This exception is already handled in the outer catch block.

Change-Id: Ifc04b8e138732a97b9c0ee323af923b8e52a7f8e
2016-07-05 12:57:58 -07:00
Shawn Pearce 306932c701 ReceivePack: enable capabilities immediately on first line
Instead of deferring until after command parsing, enable the
capabilities after the first pkt-line has been read from the client.
This allows the server to setup the side-band-64k channel immediately.

Change-Id: I141b7fc92e983a41d3a58da8e1464a6917422b6c
2016-07-05 12:37:26 -07:00
Shawn Pearce 60ca93798c ReceivePack: Use Java 7 multi-catch for unpackError
Change-Id: I328f5952f23185a7d8f81a144979b4f8a095bd60
2016-07-05 12:35:26 -07:00
Shawn Pearce ae5870a1f2 PushConnectionTest: Use Java 7 try-with-resources
Change-Id: I429066d2af9c5fb8bf8bc038b99ebe5a7dc09170
2016-07-05 12:32:24 -07:00
Shawn Pearce b57d060408 push: Report fatal server errors during pack writing
If the push client has requested side-band support the server can
signal a fatal error parsing the pack using the error channel (3)
and then hang up. This may cause the PackWriter to fail to write to
data onto the network socket, which throws a misleading error back
up to the application and the user.

During a write failure poll the input to see if the side band system
can parse out an error message off channel 3. This should be fast as
there will either be an error present in the buffer, or the remote will
also have hung-up on the side band channel. In the case of a hang-up
just rethrow the original IOException as its a network error.

This roughly matches what C git does; once commands are sent and the
packer is started a new thread runs in the background to decode any
possible server error during unpacking on the remote peer

Change-Id: Idb37a4a122a565ec4b59130e08c27d963ba09395
2016-07-05 09:18:41 -07:00
Shawn Pearce 61d444305e ReceivePack: Catch InvalidObjectIdException instead of IAE
The more specific type InvalidObjectIdException is thrown by
ObjectId.fromString().  Use it here in ReceivePack as the more
generic IAE is never thrown by the body of the try-catch block.

Change-Id: I53fc13c561c7d429a50b5eb82773f1a670431c54
2016-07-05 09:18:41 -07:00
Shawn Pearce 5196798cb7 UploadPack: Include peeled ObjectId as advertised
A RefAdvertiser writing to the network includes both the reference's
ObjectId and its peeled ObjectId in the advertised set.  In smart HTTP
negotiation requests may bypass the RefAdvertiser and quickly build
the set based on current refs; include the peeled ObjectIds to match
behavior with the normal bidirectional protocols on git:// and SSH.

Change-Id: I5371bed60da36e8d12c4ad9a5c1d91a0f0ad486b
2016-07-05 09:18:41 -07:00