I don't want to play games with the order of release here, its
probably safer to release the reader before the database, just
in case the one depends on the other.
Change-Id: I2394c7d2477eaf7a7e1556fc3393c59d3b31e764
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
By doing the release at the higher level class, we can ensure
the release occurs if the inserter was allocated, even if the
implementation forgets to do this. Since the higher level class
is what allocated it, it makes sense to have it also do the release.
Change-Id: Id617b2db864c3208ed68cba4eda80e51612359ad
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Everyone else does. This must have been a spot I missed during
some sort of squash while developing the series.
Change-Id: I62eae50b618f47ee33ad7cf71fc05b724f603201
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Similar to what we did with the file code, move the pack writer
into its own package so the related classes and their package
private methods are hidden from the rest of the library.
Change-Id: Ic1b5c7c8c8d266e90c910d8d68dfc8e93586854f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We no longer need an ObjectLoader to be lazy and try to delay
the materialization of the object content. That was done only
to support PackWriter searching for a good reuse candidate.
Instead, simplify the code base by doing the materialization
immediately when the loader asks for it, because any caller
asking for the loader is going to need the content.
Change-Id: Id867b1004529744f234ab8f9cfab3d2c52ca3bd0
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
These were only used by PackWriter to help it filter object
representations. Their only user disappeared when we rewrote the
object selection code path to use the new representation type.
Change-Id: I9ed676bfe4f87fcf94aa21e53bda43115912e145
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Rather than making a loader, and then using that to fill the object
representation, parse the header and set up our data directly.
This saves some time, as we don't waste cycles on information we
won't use right now.
The weight computed for a representation is now its actual stored
size in the pack file, rather than its inflated size. This accounts
for changes made when the compression level is modified on the
repository. It is however more costly to determine the weight of
the object, since we have to find its length in the pack. To try and
recover that cost we now cache the length as part of our ObjectToPack
record, so it doesn't have to be found during the output phase.
A LocalObjectToPack now costs us (assuming 32 bit pointers):
(32 bit) (64 bit)
vm header: 8 bytes 8 bytes
ObjectId: 20 bytes 20 bytes
PackedObjectInfo: 12 bytes 12 bytes
ObjectToPack: 8 bytes 12 bytes
LocalOTP: 20 bytes 24 bytes
----------- ---------
68 bytes 74 bytes
Change-Id: I923d2736186eb2ac8ab498d3eb137e17930fcb50
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This move isolates all of the local file specific implementation code
into a single package, where their package-private methods and support
classes are properly hidden away from the rest of the core library.
Because of the sheer number of files impacted, I have limited this
change to only the renames and the updated imports.
Change-Id: Icca4884e1a418f83f8b617d0c4c78b73d8a4bd17
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Objects that fall completely within a single window can be worked
with in a zero-copy fashion, provided that the window is backed by
a normal byte[] and not by a ByteBuffer.
This works for a surprising number of objects. The default window
size is 8 KiB, but most deltas are quite a bit smaller than that.
Objects smaller than 1/2 of the window size have a very good chance
of falling completely within a window's array, which means we can
work with them without copying their data around.
Larger objects, or objects which are unlucky enough to span over a
window boundary, get copied through the temporary buffer. We pay
a tiny penalty to realize we can't use the zero-copy code path,
but its easier than trying to keep track of two adjacent windows.
With this change (as well as everything preceeding it), packing
is actually a bit faster. Some crude benchmarks based on cloning
linux-2.6.git (~324 MiB, 1,624,785 objects) over localhost using
C git client and JGit daemon shows we get better throughput, and
slightly better times:
Total Time | Throughput
(old) (now) | (old) (now)
--------------+---------------------------
2m45s 2m37s | 12.49 MiB/s 21.17 MiB/s
2m42s 2m36s | 16.29 MiB/s 22.63 MiB/s
2m37s 2m31s | 16.07 MiB/s 21.92 MiB/s
Change-Id: I48b2c8d37f08d7bf5e76c5a8020cde4a16ae3396
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Output of selected reuses is refactored to use a new ObjectReuseAsIs
interface that extends the ObjectReader. This interface allows the
reader to control how it performs the reuse into the output stream,
but also allows it to throw an exception to request the writer to
find a different candidate representation.
The PackFile reuse code was overhauled, cleaning up the APIs so they
aren't exposed in the object loader, but instead are now a single
method on the PackFile itself. The reuse algorithm was changed to do
a data verification pass, followed by the copy pass to the output.
This permits us to work around a corrupt object in a pack file by
seeking another copy of that object when this one is bad.
The reuse code was also optimized for the common case, where the
in-pack representation is under 16 KiB. In these smaller cases
data is sent to the pack writer more directly, avoiding some copying.
Change-Id: I6350c2b444118305e8446ce1dfd049259832bcca
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The new selection implementation uses a public API on the
ObjectReader, allowing the storage library to enumerate its
candidates and select the best one for this packer without
needing to build a temporary list of the candidates first.
Change-Id: Ie01496434f7d3581d6d3bbb9e33c8f9fa649b6cd
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Make the lower bits available for flags that PackWriter can use to
keep track of facts about the object. We shouldn't need more than
2^24 delta depths, unpacking that chain is unfathomable anyway.
This change gets us 4 bits that are unused in the lower end of the
word, which are typically easier to load from Java and most machine
instruction sets. We can use these in later changes.
Change-Id: Ib9e11221b5bca17c8a531e4ed130ba14c0e3744f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The ObjectReader class is dual-purposed into being a factory for the
ObjectToPack, permitting specific ObjectDatabase implementations
to override the method and offer their own custom subclass of the
generic ObjectToPack class. By allowing them to directly extend the
type, each implementation can add custom fields to support tracking
where an object is stored, without incurring any additional penalties
like a parallel Map<ObjectId,Object> would cost.
The reader was chosen to act as a factory rather than the database,
as the reader will eventually be tied more tightly with the
ObjectWalk and TreeWalk. During object enumeration the reader
would have had to load the object for the RevWalk, and may chose
to cache object position data internally so it can later be reused
and fed into the ObjectToPack instance supplied to the PackWriter.
Since a reader is not thread-safe, and is scoped to this PackWriter
and its internal ObjectWalk, its a great place for the database to
perform caching, if any.
Right now this change goes a bit backwards by changing what should
be generic ObjectToPack references inside of PackWriter to the very
PackFile specific LocalObjectToPack subclass. We will correct these
in a later commit as we start to refine what the ObjectToPack API
will eventually look like in order to better support the PackWriter.
Change-Id: I9f047d26b97e46dee3bc0ccb4060bbebedbe8ea9
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This shortens the implementation within PackWriter, and starts
to open the door for some other refactorings based on changing
the ObjectToPack to be a public part of the API.
Change-Id: Id849cbffc4de20b903e844a2de7737eeb8b7a3ff
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some types of repositories might not be stored on local disk. For
these, they will most likely return null for getDirectory() as the
java.io.File type cannot describe where their storage is, its not
in the host's filesystem.
Document that getDirectory() can return null now, and update all
current non-test callers in JGit that might run into problems on
such repositories. For the most part, just act like its bare.
Change-Id: I061236a691372a267fd7d41f0550650e165d2066
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Replace the old crude event listener system with a much more generic
implementation, patterned after the event dispatch techniques used
in Google Web Toolkit 1.5 and later.
Each event delivers to an interface that defines a single method,
and the event itself is what performs the delivery in a type-safe
way through its own dispatch method.
Listeners are registered in a generic listener list, indexed by
the interface they implement and wish to receive an event for.
Delivery of events is performed by looping through all listeners
implementing the event's corresponding listener interface, and using
the event's own dispatch method to deliver the event. This is the
classical "double dispatch" pattern for event delivery.
Listeners can be unregistered by invoking remove() on their
registration handle. This change therefore requires application
code to track the handle if it wishes to remove the listener at a
later point in time.
Event delivery is now exposed as a generic public method on the
Repository class, making it easier for any type of message to
be sent out to any type of listener that has registered, without
needing to pre-arrange for type-safe fireFoo() methods.
New event types can be added in the future simply by defining a
new RepositoryEvent subclass and a corresponding RepositoryListener
interface that it dispatches to. By always adding new events through
a new interface, we never need to worry about defining an Adapter
to provide default no-op implementations of new event methods.
Change-Id: I651417b3098b9afc93d91085e9f0b2265df8fc81
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This better matches with the name used in the environment
(GIT_WORK_TREE), in the configuration file (core.worktree),
and in our builder object.
Since we are already breaking a good chunk of other code
related to repository access, and this fairly easy to fix
in an application's code base, I'm not going to offer the
wrapper getWorkDir() method.
Change-Id: Ib698ba4bbc213c48114f342378cecfe377e37bb7
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The new FileRepositoryBuilder class helps applications to construct
a properly configured FileRepository, with properties assumed based
upon the standard Git rules for the local filesystem.
To better support simple command line applications, environment
variable handling and repository searching was moved into this
builder class.
The change gets rid of the ever-growing FileRepository constructor
variants, and the multitude of java.io.File typed parameters,
by using simple named setter methods.
Change-Id: I17e8e0392ad1dbf6a90a7eb49a6d809388d27e4c
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Not every type of Repository will be able to map an ObjectId into
a local file system path that stores that object's file contents.
Heck, its not even true for the FileRepository, as an object can
be stored in a pack file and not in its loose format.
Remove this from our public API, it was a mistake to publish it.
Change-Id: I20d1b8c39104023936e6d46a5b0d7ef39ff118e8
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Rather than relying on the repository's ability to give us the
local file path for a loose object, just pass its inflated form to
the ObjectInserter for the repository. We have to recompress it,
which may slow down fetches, but this is the slow dumb protocol.
The extra cost to do the compression locally isn't going to be a
major bottleneck.
This nicely removes the nasty part about computing the object
identity by hand, allowing us to instead rely upon the inserter's
internal computation. Unfortunately it means we might store a loose
object whose SHA-1 doesn't match the expected SHA-1, such as if the
remote repository was corrupted. This is fairly harmless, as the
incorrectly named object will now be stored under its proper name,
and will eventually be garbage collected, as its not referenced by
the local repository.
We have to flush the inserter after the object is stored because
we aren't sure if we need to read the object later, or not.
Change-Id: Idb1e2b1af1433a23f8c3fd55aeb20575e6047ef0
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The WindowCache is an implementation detail of PackFile and how its
used by ObjectDirectory. Lets start to hide it and replace the public
API with a more generic concept, ObjectReader.
Because PackedObjectLoader is also considered a private detail of
PackFile, we have to make PackWriter temporarily dependent upon the
WindowCursor and thus FileRepository and ObjectDirectory in order to
just start the refactoring. In later changes we will clean up the
APIs more, exposing sufficient support to PackWriter without needing
the file specific implementation details.
Change-Id: I676be12b57f3534f1285854ee5de1aa483895398
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Not every object storage system will have the concept of alternate
object databases to search, and even if they do, they may not have
the notion of fast-access / slow-access split like we do within
the ObjectDirectory code for pack files and loose objects.
Push all of that down below the generic API so that it is a hidden
detail of the ObjectDirectory and its related supporting classes.
Change-Id: I54bc1ca5ff2ac94dfffad1f9a9dad7af202b9523
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some newer style APIs are updated to use the newer ObjectInserter
interface instead of the now deprecated ObjectWriter. In many of
the unit tests we don't bother to release the inserter, these are
typically using the file backend which doesn't need a release,
but in the future should use an in-memory HashMap based store,
which really wouldn't need it either.
Change-Id: I91a15e1dc42da68e6715397814e30fbd87fa2e73
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The ObjectInserter API permits ObjectDatabase implementations to
control their own object insertion behavior, rather than forcing
it to always be a new loose file created in the local filesystem.
Inserted objects can also be queued and written asynchronously to
the main application, such as by appending into a pack file that
is later closed and added to the repository.
This change also starts to open the door to non-file based object
storage, such as an in-memory HashMap for unit testing, or a more
complex system built on top of a distributed hash table.
To help existing application code port to the newer interface we
are keeping ObjectWriter as a delegation wrapper to the new API.
Each ObjectWriter instances holds a reference to an ObjectInserter
for the Repository's top-level ObjectDatabase, and it flushes and
releases that instance on each object processed.
Change-Id: I413224fb95563e7330c82748deb0aada4e0d6ace
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
A repository implementation might support storing configurations
on a non-file storage system, so widen the return value to be any
type of configuration.
Change-Id: If9a0928f4b3ef29a24d270b0ce585a6e77f6fac6
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
To support other storage models other than just the local filesystem,
we split the Repository class into a nearly abstract interface and
then create a concrete subclass called FileRepository with the file
based IO implementation.
We are using an abstract class for Repository rather than the much
more generic interface, as implementers will want to inherit a large
array of utility functions, such as resolve(String). Having these in
a base class makes it easy to inherit them.
This isn't the final home for lib.FileRepository. Future changes
will rename it into storage.file.FileRepository, but to do that we
need to also move a number of other related class, which we aren't
quite ready to do.
Change-Id: I1bd54ea0500337799a8e792874c272eb14d555f7
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If the working tree isn't available, it doesn't make any sense to
obtain the merge heads, or the buffered commit message. The
repository shouldn't have a partial merge state to read. Throw back
the same exception we do when invoking getWorkDir() on a bare
repository instance.
Change-Id: I762c55890b7fe272a183da583f910671d1cadf71
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This permits us to leave the implementation of these methods here in
the Repository class, but later refactor how the directory is accessed
into a subclass.
Change-Id: I5785b2009c5b7cca0fb070a968e50814ce847076
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
A bare repository cannot be checked out, committed to, etc. as it
doesn't have a working directory. Define this as a state since the
state enumeration exists only to describe how a working directory
can be modified.
Change-Id: I0a299013c6e42fef6cae3f6a9446f8f6c8e0514a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This better matches with the other configuration variable,
'userConfig', and helps to make it clear what config object
we are dealing with.
Change-Id: I2c585649aecc805e8e66db2f094828cd2649e549
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Change the Repository API to use straight-up FileBasedConfig.
This lets us remove the subclass RepositoryConfig and stop having
a specialized configuration type for repository, letting us instead
focus the config type heirarchy on type-of-storage rather than use.
Change-Id: I7236800e8090624453a89cb0c7a9a632702691c6
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Instead of using the internal field directly to access references
or objects, use the getter method to obtain the proper type of
database, and follow down from there. This permits us to later
do a refactoring that makes those methods abstract and strips the
field out of the Repository class, moving it into a concrete base
class that is more storage implementation specific.
Change-Id: Ic21dd48800e68a04ce372965ad233485b2a84bef
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This method doesn't need to be synchronized, as its only a proxy to
create(boolean), which is the real worker. While we are touching
it try to improve the Javadoc and whitespace nearby.
Change-Id: Ibdddec6e518ca6d7439cfad90fedfcdc2d6b7a2e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This permits the repository implementation to know what its
alternates concept means, and avoids needing to expose finer details
about the ObjectDatabase to network code like the RefAdvertiser.
Change-Id: Ic6d173f300cb72de34519c7607cf7b0ff3ea6882
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
In the future getDirectory() can return null. Avoid an NPE here by
refactoring the code to support conditionally skipping a check for
the properties file in the repository directory, falling to only
the user's ~/ file location.
Change-Id: I76f5503d4063fdd9d24b7c1b58e1b09ddf1a5670
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If the destination repository doesn't use an ObjectDirectory to
store its objects, we can't download to the object directory.
Instead pull the pack-*.idx files down to temporary files in the
JVM's default temporary directory.
Change-Id: Ied16bc89be624d87110ba42ba52d698a6ea7d982
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When reading or locking the index of a repository, we need to use
the index file specified by the repository, to ensure we correctly
honor what the repository was configured with.
Change-Id: I5be366ce32d7923b888dc01d19335912b01b7c4c
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Its not strictly required that we sort topologically in order to
produce a valid pack file. This was just something that Linus
thought would be a good idea to do. In practice its not that
important for most repositories. Local file IO quickly falls
out of the pattern that topological sorting provides any sort
of benefit for, so expending extra resources to enforce it when
we make a pack isn't really worth it.
I'm removing this sort in the pipeline because later changes
would support really efficient COMMIT_TIME_DESC sorting on a
non-file storage system, but TOPO sorting would be a bit more
ugly to run, due to the in-memory delays it imposes.
Change-Id: I0121453461c2140c6917cb10c6df584eb47e5795
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If UploadPack invokes flush() on the output stream we pass it, its
most likely the progress messages coming down the side band stream.
As pack generation can take a while, we want to push that down
at the client as early as we can, to keep the connection alive,
and to let the user know we are still working on their behalf.
Ensure we dump the temporary buffer whenever flush() is invoked,
otherwise the messages don't get sent in a timely fashion to the
user agent (in this case, git fetch).
We specifically don't implement flush() for ReceivePack right now,
as that protocol currently does not provide progress messages to
the user, but it does invoke flush several times, as the different
streams include '0000' type flush-pkts to denote various end points.
Change-Id: I797c90a2c562a416223dc0704785f61ac64e0220
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We want to eventually get rid of the mapCommit, mapTree APIs on
Repository and force everyone into the faster parsers that exist
in RevWalk. Rewriting resolve in terms of the faster parsers is
a good first step.
It actually simplifies the code a bit, as we no longer need to
keep track of an ObjectId and an Object (the parsed form), since
all RevObjects implicitly have their ObjectId readily available.
Change-Id: I4d234630195616e2c263e7e70038b55a1be4e7a3
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Instead of peeling things by hand in application level code, defer
the peeling logic into RevWalk's new peel utility method.
Change-Id: Idabd10dc41502e782f6a2eeb56f09566b97775a8
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We already have these objects parsed and cached in our object pool.
We shouldn't be looking them up via the legacy mapObject API, but
instead can use the pool and the faster parsing routines available
through the RevWalk that we extend.
While we are here fixing the code, lets also correct the tag date
sorting to accept tags that have no tagger identity, because they
were created before Git knew how to store that field.
Change-Id: Id49a11f6d9c050c82b876e5e11058840c894b2d7
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Rather than relying on the helpers in RepositoryConfig to get
these objects, obtain them directly through the Config API.
Its only slightly more verbose, but permits us to work with the
base Config class, which is more flexible than the highly file
specific RepositoryConfig.
This is what I really meant to do when I added the section parser
and caching support to Config, we just failed to finish updating
all of the call sites.
Change-Id: I481cb365aa00bfa8c21e5ad0cd367ddd9c6c0edd
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We don't have to assume/depend on RepositoryConfig here, these
two tests can use higher level versions of the class and still
come up with the same test. That frees us up to do some changes
to the RepositoryConfig API.
Change-Id: Ia7b263c8c5efa3fae1054416d39c546867288132
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Under smart HTTP the biDirectionalPipe flag is false, and we return
back immediately at this point in the negotiation process. There is
no need to flush the stream to the client, the request is over and
it will be automatically flushed out by the higher level servlet
that invoked us. Avoiding flush here allows us to only use flush
after a progress message is sent during pack generation.
Change-Id: Id0c8b7e95e3be6ca4c1b479e096bed6b0283b828
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This simplifies the PackIndex code, which is trying to quickly copy
an existing ObjectId into a MutableObjectId. Rather than having
the PackIndex violate the ObjectId's internals, expose a copy from
function similar to the other ones for copying from raw byte arrays
or hex formatted strings.
Change-Id: I142635cbece54af2ab83c58477961ce925dc8255
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Storage systems can use these implementations to compare a passed
AnyObjectId with a stored representation of an ObjectId in the
canonical network byte order format. This can be useful to do a
binary search, or just linear scan, over an encoded storage file.
Change-Id: I8c72993c4f4c6e98d599ac2c9867453752f25fd2
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
An implementation might prefer to use the RefList type here, and
RefList is part of our public API. Expose the constructor so callers
who have a RefList can take advantage of the existing sorting.
Change-Id: I545867f85aa2c479d2d610024ebbe318144709c8
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When we finally move RefDirectory to the new storage.file package,
its associated RefDirectoryUpdate will need visiblity to this
constructor in order to initialize itself. This is true of any
other repository implementation, so make it protected rather than
package level visible.
Change-Id: If838aec9baeb80ee2f12dcbca717657c725a9242
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Repository implementations outside of .lib need to be able to
create these events and deliver them to listening application code.
Expose and document the constructors so that they are visible when
we move FileRepository into storage.file.FileRepository.
Change-Id: I7fb6e8f4f5fdab683c5ebb5267673aa6d5b560bb
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
A Git reference name must never end with ".lock", as it would
confuse any existing C client that tries to obtain a clone of the
repository over the network. Even if the repository isn't on a
local filesystem, it still should ban that suffix.
Because I plan to move LockFile to storage.file and make it a private
implementation detail of the local file system storage model,
we can't rely on its package level SUFFIX field here. Making it
public probably won't work long-term either, as I also plan to
pull storage.file into its own separate project that depends on
the core library.
So, just inline the constant here. Its as foribidden as ":" is.
Change-Id: If85076861baeacc183b82696375a13e935ba8836
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Modifications to various classes in order to allow serialization
for use of JGit in Hudson's git plugin.
Change-Id: If088717d3da7483538c00a927e433a74085ae9e6
If a repository is "bare", it currently still returns a working directory.
This conflicts with the specification of "bare"-ness.
Bug: 311902
Change-Id: Ib54b31ddc80b9032e6e7bf013948bb83e12cfd88
Signed-off-by: Mathias Kinzler <mathias.kinzler@sap.com>
Currently, there is no way to read the content
of the Git Configuration in a way that would
allow to list all configured values generically.
This change extends the Config class in such a
way as to being able to get a list of sections and
to get a list of names for any given section or
subsection.
This is required in able to implement proper
configuration handling in EGit (show all the
content of a given configuration similar to
"git config -l").
Change-Id: Idd4bc47be18ed0e36b11be8c23c9c707159dc830
Signed-off-by: Mathias Kinzler <mathias.kinzler@sap.com>
Refactor and extend the internals so users can override and
intervene during formatting, e.g. to colorize output.
Change-Id: Ia1cf40cfd4a5ed7dfb6503f8dfc617237bee0659
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
On Windows, FS_Win32_Cygwin has been used if a Cygwin Git installation
is present in the PATH. Assuming that the user works with the Cygwin
Git installation may result in unnecessary overhead if he actually
does not.
Applications built on top of jgit may have more knowledge on the
actually used Git client (Cygwin or not) and hence should be able to
configure which FS to use accordingly.
Change-Id: Ifc4278078b298781d55cf5421e9647a21fa5db24
A Change-Id helps tools like Gerrit Code Review to keeps different
versions of a patch together. The Change-Id is computed as a SHA-1
hash of some of the same basic information as a commit id on the first
commit intended to solve a particular problem and then reused for
updated solutions.
Change-Id: I04334f84e76e83a4185283cb72ea0308b1cb4182
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
The J2SE NIO APIs require that FileChannel close the underlying file
descriptor if a thread is interrupted while it is inside of a read or
write operation on that channel. This is insane, because it means we
cannot share the file descriptor between threads. If a thread is in
the middle of the FileChannel variant of IO.readFully() and it
receives an interrupt, the pack will be automatically closed on us.
This causes the other threads trying to use that same FileChannel to
receive IOExceptions, which leads to the pack getting marked as
invalid. Once the pack is marked invalid, JGit loses access to its
entire contents and starts to report MissingObjectExceptions.
Because PackWriter must ensure that the chosen pack file stays
available until the current object's data is fully copied to the
output, JGit cannot simply reopen the pack when its automatically
closed due to an interrupt being sent at the wrong time. The pack may
have been deleted by a concurrent `git gc` process, and that open file
descriptor might be the last reference to the inode on disk. Once its
closed, the PackWriter loses access to that object representation, and
it cannot complete sending the object the client.
Fortunately, RandomAccessFile's readFully method does not have this
problem. Interrupts during readFully() are ignored. However, it
requires us to first seek to the offset we need to read, then issue
the read call. This requires locking around the file descriptor to
prevent concurrent threads from moving the pointer before the read.
This reduces the concurrency level, as now only one window can be
paged in at a time from each pack. However, the WindowCache should
already be holding most of the pages required to handle the working
set for a process, and its own internal locking was already limiting
us on the number of concurrent loads possible. Provided that most
concurrent accesses are getting hits in the WindowCache, or are for
different repositories on the same server, we shouldn't see a major
performance hit due to the more serialized loading.
I would have preferred to use a pool of RandomAccessFiles for each
pack, with threads borrowing an instance dedicated to that thread
whenever they needed to page in a window. This would permit much
higher levels of concurrency by using multiple file descriptors (and
file pointers) for each pack. However the code became too complex to
develop in any reasonable period of time, so I've chosen to retrofit
the existing code with more serialization instead.
Bug: 308945
Change-Id: I2e6e11c6e5a105e5aef68871b66200fd725134c9
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Merges the current head with one other commit.
In this first iteration the merge command supports
only fast forward and already up-to-date.
Change-Id: I0db480f061e01b343570cf7da02cac13a0cbdf8f
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
The CommitCommand should take care to create a merge commit if the file
$GIT_DIR/MERGE_HEAD exists. It should then read the parents for the merge
commit out of this file. It should also take care that when commiting
a merge and no commit message was specified to read the message from
$GIT_DIR/MERGE_MSG.
Finally the CommitCommand should remove these files if the commit
succeeded.
Change-Id: I4e292115085099d5b86546d2021680cb1454266c
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
The strings are externalized into the root resource bundles.
The resource bundles are stored under the new "resources" source
folder to get proper maven build.
Strings from tests are, in general, not externalized. Only in
cases where it was necessary to make the test pass the strings
were externalized. This was typically necessary in cases where
e.getMessage() was used in assert and the exception message was
slightly changed due to reuse of the externalized strings.
Change-Id: Ic0f29c80b9a54fcec8320d8539a3e112852a1f7b
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
In close() method of SshFetchConnection and SshPushConnection
errorThread.join() can wait forever if JSch will not close the
channel's error stream. Join with a timeout, and interrupt the
copy thread if its blocked on data that will never arrive.
Bug: 312863
Change-Id: I763081267653153eed9cd7763a015059338c2df8
Reported-by: Dmitry Neverov <dmitry.neverov@gmail.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If we get an interrupt during an IO operation (src.read or dst.write)
caused by the flush() method incrementing the flush counter, ensure
we restart the proper section of code. Just ignore the interrupt
and continue running.
Bug: 313082
Change-Id: Ib2b37901af8141289bbac9807cacf42b4e2461bd
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The number of bytes to copy was truncated to an int, but the
pack's copyToStream() method expected to be passed a long here.
Pass through the long so we don't truncate a giant object.
Change-Id: I0786ad60a3a33f84d8746efe51f68d64e127c332
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Rather than holding onto the PackedObjectLoader, only hold the
PackFile and the object offset. During a reuse copy that is all
we should need to complete a reuse, and the other parts of the
PackedObjectLoader just waste memory.
This change reduces the per-object memory usage of a PackWriter by
32 bytes on a 32 bit JVM using only OFS_DELTA formatted objects.
The savings is even larger (by another 20 bytes) for REF_DELTAs.
This is close to a 50% reduction in the size of ObjectToPack,
making it rather worthwhile to do.
Beyond the memory reduction, this change will help to make future
refactoring work easier. We need to redo the API used to support
copying data, and disconnecting it from the PackedObjectLoader is
a good first step.
Change-Id: I24ba4e621e101f14e79a16463aec5379f447aa9b
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Rather than keep track of both the position of the object, and the
position of its data, just keep track of the number of bytes used
by the object's header in the pack. This shaves 4 bytes out of the
size of the PackedObjectLoader instances.
We also can defer the addition instruction to the materialize()
operation, avoiding it entirely if the caller never actually uses
the loader. This may be relevant for PackWriter invocations,
where only 1 loader gets chosen for a given object, even though
the object may appear on disk in more than one pack file.
Error reporting is now simplified, as we can rely on the object
offset rather than its data offset. This is the value displayed
by pack debugging tools like `git verify-pack -v`, so its better
to use that in our own errors.
Because nobody needs getDataOffset() now, we can drop that from
the public API.
Change-Id: Ic639c0d5a722315f4f5c8ffda6e26643d90e5f42
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Since we use this code twice, pull it into a private method. Let
the compiler/JIT worry about whether or not this logic should be
inlined into the call sites.
Change-Id: Ia44fb01e0328485bcdfd7af96835d62b227a0fb1
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Originally when I wrote this code I had hoped to use OffsetCache
to also implement the UnpackedObjectCache. But it turns out they
need rather different code, and it just wasn't worth trying to
reuse the OffsetCache base class.
Before doing any major refactoring or code cleanups here, squash the
two classes together and delete OffsetCache. As WindowCache is our
only subclass, this is pretty simple to do. We also get a minor
code reduction due to less duplication between the two classes,
and the JIT should be able to do a better job of optimization here
as we can define types up front rather than relying on generics
that erase back to java.lang.Object.
Change-Id: Icac8bda01260e405899efabfdd274928e98f3521
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When we read the object header we copy 20 bytes from the pack data,
then start parsing out the type and the inflated size. For most
objects, this is only going to require 3 bytes, which is sufficient
to represent objects with inflated sizes of up to 2^16. The local
buffer however still has 17 bytes remaining in it, and that can be
used to satisfy the OBJ_OFS_DELTA header.
We shouldn't need to worry about walking off the end of the buffer
here, because delta offsets cannot be larger than 64 bits, and that
requires only 9 bytes in the OFS_DELTA encoding.
Assuming worst-case scenarios of 9 bytes for the OFS_DELTA encoding,
the pack file itself must be approaching 2^64 bytes, an infeasible
size to store on any current technology. However, even if this
were the case we still have 11 bytes for the type/size header.
In that encoding we can represent an object as large as 2^74 bytes,
which is also an infeasible size to process in JGit.
So drop the second read here.
The data offsets we pass into the ObjectLoaders being constructed
need to be computed individually now. This saves a local variable,
but pushes the addition operation into each branch of the switch.
Change-Id: I6cf64697a9878db87bbf31c7636c03392b47a062
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
JSch may hang or abort with the timeout if JGit connects before
its obtained the streams. Instead defer the connect() call until
after the streams have been configured.
Bug: 312383
Change-Id: I7c3a687ba4cb69a41a85e2b60d381d42b9090e3f
Reported-by: Dmitry Neverov <dmitry.neverov@gmail.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If a flush() gets delivered at the same time that we are blocking
while writing to an interruptable stream, the copy thread will
abort assuming its a stream error. Instead ignore the interrupt,
and retry the write.
Change-Id: Icbf62d1b8abe0fabbb532dbee088020eecf4c6c2
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
It is possible to miss flush() invocation in StreamCopyThread.
In this case some data will not be sent to remote host and we will
wait forever (or until timeout) in src.read().
Use a counter to keep track of the flush requests.
Change-Id: Ia818be9b109a1674d9e2a9c78e125ab248cfb75b
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Added a new package org.eclipse.jgit.api and a builder-style API for
jgit. Added also the first implementation for two git commands: Commit
and Log.
This API is intended to be used by external components when
functionalities of the standard git commands are required. It will also
help to ease writing JGit tests.
For internal usages this API may often not be optimal because the git
commands are doing much more than required or they expect parameters of
an unappropriate type.
Change-Id: I71ac4839ab9d2f848307eba9252090c586b4146b
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Currently, if the Index contains a file in more than one stage, only
the last entry (containing the highest stage) will be registered in
GitIndex. For applications it can be useful to not only know about the
highest stage, but also which other stages are present, e.g. to detect
the type of conflict the file is in.
Change-Id: I2d4ff9f6023335d9ba6ea25d8e77c8e283ae53cb
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
The repository state tells in which state the repo is and also which actions
are currently allowed. The state MERGING is telling that a commit is not
possible. But this is only true in the case of unmerged paths in the index.
When we are merging but have resolved all conflicts then we are in a special
state: We are still merging (means the next commit should have multiple
parents) but a commit is now allowed.
Since the MERGING state "canCommit()" cannot be enhanced to return true/false
based on the index state (MERGING is an enum value which does not have a
reference to the repository its state it is representing) I had to introduce a new
state MERGING_RESOLVED. This new state will report that a commit is possible.
CAUTION: there might be the chance that users of jgit previously blindly did a
plain commit (with only one parent) when the RepositoryState allowed them to
do so. With this change these users will now be confronted with a RepositoryState
which says a commit is possible but before they can commit they'll have to
check the MERGE_MESSAGE and MERGE_HEAD files and use the info from these
files.
Change-Id: I0a885e2fe8c85049fb23722351ab89cf2c81a431
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If two keys are the same length, but don't share the same sequence
of characters, we were incorrectly claiming they still matched due
to a bug in the for loop condition. I used the wrong variable and
the loop never executed, resulting in equality anytime the two keys
being compared were the same length.
Use the proper local variable to loop through the arrays, and add
a JUnit test to verify equality works as expected.
Change-Id: I4a02400e65a9b2e0da925b05a2cc4b579e1dd33a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If a loose object was corrupted by truncation, JGit would hang.
Change-Id: I7e4c14f44183a5fcb37c1562e81682bddeba80ad
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
If a concurrent thread picks up a newly created PackFile and adds
it to the pack list before the IndexPack thread itself can insert
the item onto the front of the list, do nothing and use the item
that was picked up by that other concurrent scanning thread.
This avoids a potential condition where the same pack exists in
memory twice, which causes confusion later during a rescan of the
directory because we don't know exactly which PackFile instance
should be retained into the new list, and which should be discarded.
We can stop searching through the old pack list as soon as the
sort function declares that the item to insert should be before
the item already in the list. Because the list is always sorted
by modification time (in seconds), we should never encounter a
case where the pack is positioned at the wrong spot in the list.
This early break out still permits an efficient implementation of
the common case, inserting a new pack at the head of the list.
Change-Id: Ice4459bbd4ee9487078aff5257893883d04f05fb
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
There is a potential race condition during insertPack that can lead
to us having the same pack file open twice in the same directory.
A different thread can miss an object on disk, and trigger a scan
of the directory, and notice the pack that was put in by IndexPack.
So the pack winds up in the newly created PackList.
The IndexPack thread then wakes up and finishes its insertPack by
creating a new PackFile and inserting it into position 0 of the list.
We now have the same pack listed twice.
Readers will favor the earlier PackFile instance, because its the
first one they come across as they iterate through the list.
Keep that earlier one when we scan the pack directory again, as
this will avoid needing to purge out all of the windows that may
have been cached.
Of course we should also fix that race condition, but this block
was taking the wrong resolution if this error ever shows up, so
lets first fix the block to use a more sane resolution.
Change-Id: I0d339b9fd1dd8012e8fe5a564b893c0f69109e28
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This reuse line was identical between the two branches related to
reusing a delta, or reusing a whole object. Either way they reuse
the body of the object as-is. So just make that a common function
after the header is written.
Change-Id: I0e6673b8e813c8c08c594ea2ba546fd366339d5d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If a corrupt loose object is read, UnpackedObjectLoader was disposing
of the Inflater, and then attempting to return the disposed Inflater
to the InflaterCache. Since the disposed Inflater had its native
libz resource deallocated and its reference cleared out, the Inflater
threw NullPointerException and refused to reset itself before being
put back into the cache.
Instead of disposing of the Inflater when corruption is found, do
nothing, and allow it to be returned to the cache. The instance
will get reset, and should be usable by a future caller.
Bug: 310291
Change-Id: I44f2247c08b6e04fa62f8399609341b07508c096
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
* receive-pack-filter:
ReceivePack: Clarify the check reachable option
ReceivePack: Micro-optimize object lookup when checking connectivity
ReceivePack: Correct type of not provided object
IndexPack: Tighten up new and base object bookkeeping
ReceivePack: Remove need new,base object id properties
ReceivePack: Discard IndexPack as soon as possible
ReceivePack: fix ensureProvidedObjectsVisible on thin packs
Change-Id: I4ef2fcb931f3219872e0519abfcee220191d5133
This option was mis-named from day 1. Its not checking that the
objects provided by the client are reachable, its actually doing
a scan to prove that objects referenced by the client are already
reachable through another reference on the server, or were sent
as part of the pack from the client.
Rename it checkReferencedObjectsAreReachable, since we really are
trying to validate that objects referenced by the client's actions
are reachable to the client.
We also need to ensure we run checkConnectivity() anytime this is
enabled, even if the caller didn't turn on fsck for object formats.
Otherwise the check would be completely bypassed.
Change-Id: Ic352ddb0ca8464d407c6da5c83573093e018af19
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If we are checking the visibility of everything referenced in the
pack that isn't already reachable by a reference, it needs to be
in the provided set. Since the provided set lists everything that
is in this pack, we can avoid checking to see if the blob exists
on disk, because we know it should be there, it was found in the
pack we just consumed.
Change-Id: Ie3c7746f734d13077242100a68e048f1ac18c34a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If a tree was referenced but not provided in the pack, report it
as a missing tree and not as a missing blob.
Change-Id: Iab05705349cdf0d30cc3f8afc6698a8d2a941343
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The only current consumer of these collections is ReceivePack,
where it needs to test ObjectId equality between a RevObject and an
ObjectId. There we were copying from a traditional HashSet<ObjectId>
into an ObjectIdSubclassMap<ObjectId>, as the latter can perform
hashing using ObjectId's native value support, bypassing RevObject's
override on hashCode() and equals(). Instead of doing that copy,
directly create ObjectIdSubclassMap instances inside of ReceivePack.
We also only need to record the objects that do not appear in the
incoming pack, and were therefore copied from the local repositiory
in order to complete delta resolution. Instead of listing everything
that used an OBJ_REF_DELTA format, list only the objects that we
pulled from the destination repository via a normal ObjectLoader.
ReceivePack can now discard the IndexPack object, and all of its
other data, as soon as these collections are held by the check
connectivity method. This frees up memory for the ObjectWalk's
own RevObject pool.
Change-Id: I22ef71b45c2045a0202e7fd550a770ee1f6f38a6
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
These are more like internal implementation details of how IndexPack
works with ReceivePack to validate the incoming object stream.
Callers who are embedding the ReceivePack logic in their own
application don't really need to know the details of which objects
were used for delta bases in the incoming thin pack, or exactly
which objects were newly transmitted.
Hide these from the API, as exposing them through ReceivePack was
an early mistake.
Change-Id: I7ee44a314fa19e6a8520472ce05de92c324ad43e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The IndexPack object carries a good bit of state within itself about
the objects received over the wire. The earlier we can discard it,
the sooner the GC is able to reclaim this chunk of memory for other
uses. So drop it as soon as we are certain the pack is valid and we
have no connectivity concerns.
Change-Id: I1e8bc87c2e9183733043622237a064e55957891f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If ensureProvidedObjectsVisible is enabled we expected any trees or
blobs directly reachable from an advertised reference to be marked
with UNINTERESTING. Unfortunately ObjectWalk doesn't bother setting
this until the traversal is complete. Even then it won't necessarily
set it on every tree if the corresponding commit wasn't popped.
When we are going to check the base objects for the received pack,
ensure the UNINTERESTING flag gets carried into every immediately
reachable tree or blob, because these are the ones that the client
might try to use as delta bases in a thin pack.
Change-Id: I5d5fdcf07e25ac9fc360e79a25dff491925e4101
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The Iterator contract says next() shall throw NoSuchElementException
if there are no more items remaining in the iteration. We got this
wrong when I originally wrote the implementation, so fix it.
Change-Id: Iea25e6569ead5c8b3128b8a368c5b2caebec7ecc
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This class behaves like a cross between a Set and a Map, sometimes
we might expect to use the method isEmpty() to test for size() == 0.
So implement it, reducing the surprise folks get when they are given
one of these objects.
Change-Id: I0d68e1243da8e62edf79c6ba4fd925f643e80a88
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If we need to append less than 20 bytes in order to fix a thin pack
and make it complete, we need to set the length of our file back to
the actual number of bytes used because the original SHA-1 footer was
not completely overwritten. That extra data will confuse the header
and footer fixup logic when it tries to read to the end of the file.
This isn't a very common case to occur, which is why we've never
seen it before. Getting a delta that requires a whole object which
uses less than 20 bytes in pack representation is really hard.
Generally a delta generator won't make these, because the delta
would be bigger than simply deflating the whole object. I only
managed to do this with a hand-crafted pack file where a 1 byte
delta was pointed to a 1 byte whole object.
Normally we try really hard to avoid truncating, because its
typically not safe across network filesystems. But the odds of
this occurring are very low. This truncation is done on a file
we have open for writing, will append more content onto, and is
a temporary file that we won't move into position for others to
see until we've validated its SHA-1 is sane. I don't think the
truncate on NFS issue is something we need to worry about here.
Change-Id: I102b9637dfd048dc833c050890d142f43c1e75ae
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We can avoid one stat call by trying to perform a directory
listing without checking if the reference File is a directory.
Attempting a directory listing is defined to return. The other
case for null returns from list is when an I/O error occcurs.
Both cases are now intepreted as a possible plain reference. I/O
errors when reading plain references will be handled (ignored)
in scanRef().
Change-Id: I9906ed8c42eab4d6029c781aab87b3b07c1a1d2c
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
In the current implementation Repository reads user and repository
config only at creation point of time.
The new implementatiopn checks in Repository.getConfig if user or
repository config have changed on disk and reload the config if
required.
Change-Id: Ibd97515919ef66c6f8aa1a4fe8a11a6711335dad
Signed-off-by: Jens Baumgart <jens.baumgart@sap.com>
We only need to check file existense if some other stat returns
a value that may mean that the file does not exist. File.length() == 0
or File.lastModified() == 0 are two such properties. We use length
here.
Change-Id: If626b12e7bb4da994b5c086f6a5b7a12c187261c
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
When listing branches, EGit only reads the advertisement and
then disconnects. When it closes down the pack channel the remote
side is waiting for the client to send our list of commands, or a
flush-pkt to let it know there is nothing to do.
However if an error thread is open watching the SSH stderr stream,
we ask for it to finish before we send the flush-pkt. Unfortunately
the thread won't terminate until the main output stream closes,
which is waiting for the flush-pkt. A classic network deadlock.
If the output stream needs a flush-pkt we send it before we wait
for the error stream to close. If the flush-pkt is rejected, we
close down the output stream early, assuming that the remote side
is broken and we will get error information soon.
Change-Id: I8d078a339077756220c113f49d206b1bf295d434
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
By default a receive pack assumes that its user will only provide
references to objects that the user already has access to on their
local client. In certain cases, an additional check to verify the
references point only to reachable objects is necessary.
This additional checking is useful when the code doesn't trust
the client not to provide a forged SHA-1 reference to an object,
in an attempt to access parts of the DAG that they weren't allowed
to see by the configured RefFilter.
Change-Id: I3e4b8505cb2992e3e4be253abb14a1501e47b970
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
* stable-0.7:
Reuse the line buffer between strings in PacketLineIn
http.server: Use TemporaryBuffer and compress some responses
Reduce multi-level buffered streams in transport code
Fix smart HTTP client buffer alignment
Use "ERR message" for early ReceivePack problems
Catch and report "ERR message" during remote advertisements
Wait for EOF on stderr before finishing SSH channel
Capture non-progress side band #2 messages and put in result
ReceivePack: Enable side-band-64k capability for status reports
Use more restrictive patterns for sideband progress scraping
Prefix remote progress tasks with "remote: "
Decode side-band channel number as unsigned integer
Refactor SideBandInputStream construction
Refactor SideBandOutputStream to be buffered
* push-sideband:
Reuse the line buffer between strings in PacketLineIn
http.server: Use TemporaryBuffer and compress some responses
Reduce multi-level buffered streams in transport code
Fix smart HTTP client buffer alignment
Use "ERR message" for early ReceivePack problems
Catch and report "ERR message" during remote advertisements
Wait for EOF on stderr before finishing SSH channel
Capture non-progress side band #2 messages and put in result
ReceivePack: Enable side-band-64k capability for status reports
Use more restrictive patterns for sideband progress scraping
Prefix remote progress tasks with "remote: "
Decode side-band channel number as unsigned integer
Refactor SideBandInputStream construction
Refactor SideBandOutputStream to be buffered
Change-Id: Ic9689e64e8c87971f2fd402cb619082309d5587f
When reading pkt-lines off an InputStream we are quite likely to
consume a whole group of fairly short lines in rapid succession, such
as in the have exchange that occurs in the fetch-pack/upload-pack
protocol. Rather than allocating a throwaway buffer for each
line's raw byte sequence, reuse a buffer that is equal to the small
side-band packet size, which is 1000 bytes. Text based pkt-lines
are required to be less than this size because many widely deployed
versions of C Git use a statically allocated array of this length.
Change-Id: Ia5c8e95b85020f7f80b6d269dda5059b092d274d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some transports actually provide stream buffering on their own,
without needing to be wrapped up inside of a BufferedInputStream in
order to smooth out system calls to read or write. A great example
of this is the JSch SSH client, or the Apache MINA SSHD server.
Both use custom buffering to packetize the streams into the encrypted
SSH channel, and wrapping them up inside of a BufferedInputStream
or BufferedOutputStream is relatively pointless.
Our SideBandOutputStream implementation also provides some fairly
large buffering, equal to one complete side-band packet on the main
data channel. Wrapping that inside of a BufferedOutputStream just to
smooth out small writes from PackWriter causes extra data copies, and
provides no advantage. We can save some memory and some CPU cycles
by letting PackWriter dump directly into the SideBandOutputStream's
internal buffer array.
Instead we push the buffering streams down to be as close to the
network socket (or operating system pipe) as possible. This allows
us to smooth out the smaller reads/writes from pkt-line messages
during advertisement and negotation, but avoid copying altogether
when the stream switches to larger writes over a side band channel.
Change-Id: I2f6f16caee64783c77d3dd1b2a41b3cc0c64c159
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This proved to be a pretty difficult to find bug. If we read exactly
the number of response bytes from the UnionInputStream and didn't
try to read beyond that length, the last connection's InputStream is
still inside of the UnionInputStream, and UnionInputStream.isEmpty()
returns false. But there is no data present, so the next read
request to our UnionInputStream returns EOF at a point where the
HTTP client code should have started a new request in order to get
more data.
Instead of wrapping the UnionInputStream, push an dummy stream onto
the end of it which when invoked always starts the next request and
then returns EOF. The UnionInputStream will automatically pop that
dummy stream out, and then read the next request's stream.
This way we never get into the state where we don't think we need
to run another request in order to satisfy the current read request,
but we really do.
The bug was hidden for so long because BasePackConnection.init()
was always wrapping the InputStream into a BufferedInputStream
with an 8 KiB buffer. This made the odds of us reading from the
UnionInputStream the exact number of available bytes quite low, as
the BufferedInputStream would always try to read a full buffer size.
Change-Id: I02b5ec3ef6853688687d91de000a5fbe2354915d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If the application wants to, it can use sendError(String) to send one
or more error messages to clients before the advertisements are sent.
These will cause a C Git client to break out of the advertisement
parsing loop, display "remote error: message\n", and terminate.
Servers can optionally use this to send a detailed error to a client
explaining why it cannot use the ReceivePack service on a repository.
Over smart HTTP these errors are sent in a 200 OK response, and
are in the payload, allowing the Git client to give the end-user
the custom message rather than the generic error "403 Forbidden".
Change-Id: I03f4345183765d21002118617174c77f71427b5a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
GitHub broke the native git protocol a while ago by interjecting an
"ERR message" line into the upload-pack or receive-pack advertisement
list. This didn't match the expected pattern, so it caused existing
C Git clients to abort with a protocol exception.
These days, C Git clients actually look for this message and abort
with a more graceful notice to the end-user. JGit should do the
same, including setting up a custom exception type that makes it
easier for higher-level UIs to identify a message from the remote
site and present it to the user.
Change-Id: I51ab62a382cfaf1082210e8bfaa69506fd0d9786
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
JSch will allow us to close the connection and then just drop
any late messages coming over the stderr stream for the command.
This makes it easy to lose final output on a command, like from
Gerrit Code Review's post receive hook.
Instead spawn a background thread to copy data from JSch's pipe
into our own buffer, and wait for that thread to receive EOF on the
pipe before we declare the connection closed. This way we don't
have a race condition between the stderr data arriving and JSch
just tearing down the channel.
Change-Id: Ica1ba40ed2b4b6efb7d5e4ea240efc0a56fb71f6
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Any messages received on side band #2 that aren't scraped as a
progress message into our ProgressMonitor are now forwarded to a
buffer which is later included into the OperationResult object.
Application callers can use this buffer to present the additional
messages from the remote peer after the push or fetch operation
has concluded.
The smart push connections using the native send-pack/receive-pack
protocol now request side-band-64k capability if it is available
and forward any messages received through that channel onto this
message buffer. This makes hook messages available over smart HTTP,
or even over SSH.
The SSH transport was modified to redirect the remote command's
stderr stream into the message buffer, interleaved with any data
received over side band #2. Due to buffering between these two
different channels in the SSH channel mux itself the order of any
writes between the two cannot be ensured, but it tries to stay close.
The local fork transport was also modified to redirect the local
receive-pack's stderr into the message buffer, rather than going to
the invoking JVM's System.err. This gives applications a chance
to log the local error messages, rather than needing to redirect
their JVM's stderr before startup.
To keep things simple, the application has to wait for the entire
operation to complete before it can see the messages. This may
be a downside if the user is trying to debug a remote hook that is
blocking indefinitely, the user would need to abort the connection
before they can inspect the message buffer in any sort of UI built
on top of JGit.
Change-Id: Ibc215f4569e63071da5b7e5c6674ce924ae39e11
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We now advertise the side-band-64k capability inside of ReceivePack,
allowing hooks to echo status messages down the side band channel
instead of over the optional stderr stream.
This change permits hooks running inside of an http:// based push
invocation to still message the end-user with more detailed errors
than the small per-command string in the status report.
Change-Id: I64f251ef2d13ab3fd0e1a319a4683725455e5244
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
To avoid scraping a non-progress message as though it were a progress
item for the progress monitor, use a more restrictive pattern to
watch the remote side's messages. These two regexps should match
any message produced by C Git since 42e18fbf5f94 ("more compact
progress display", Oct 2007), and which first appeared in Git 1.5.4.
Change-Id: I57e34cf59d42c1dbcbd1a83dd6f499ce5e39d15d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When we pull task messages off the remote peer via sideband #2
prefix them with the string "remote: " to make it clear to the
user these are coming from the other system, and not from their
local client.
Change-Id: I02c5e67c6be67e30e40d3bc4be314d6640feb519
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This field is unsigned in the protocol, so treat it
as such when we report the channel number in errors.
Change-Id: I20a52809c7a756e9f66b3557a4300ae1e11f6d25
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Typically we refer to the raw InputStream (the stream without the
pkt-line headers on it) as rawIn, and the pkt-line header variant
as pckIn. Refactor our fields to reflect that. To ensure these
are actually the same underlying InputStream, we now create our own
PacketLineIn wrapper around the supplied raw InputStream. Its a
very low-cost object since it has only the 4 byte length buffer.
Instead of hardcoding the header length as 5, use the constant from
SideBandOutputStream. This makes it a bit more clear what we are
consuming, exactly here.
Change-Id: Iebd05538042913536b88c3ddc3adc3a86a841cc5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Instead of relying on our callers to wrap us up inside of a
BufferedOutputStream and using the proper block sizing, do the
buffering directly inside of SideBandOutputStream. This ensures
we don't get large write-throughs from BufferedOutputStream that
might overflow the configured packet size.
The constructor of SideBandOutputStream is also beefed up to check
its arguments and ensure they are within acceptable ranges for the
current side-band protocol.
Change-Id: Ic14567327d03c9e972f9734b8228178bc448867d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The tests were using a Locale.ROOT constant which was introduced
in Java 6. However, we need to retain Java 5 support.
Change-Id: I75c5648fcfc728a9aea2e839d2ad0320f5cf742f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Sasa Zivkov <sasa.zivkov@sap.com>
The initial contribution was handled through a CQ, and does not need
to be reported as an individual bug record in the project's IP log.
Its an odd corner case that the EMO IP team doesn't want to see,
even though its technically a contribution written by at least
some non-committers.
The project.skipCommit variable can now be used to mask out any
particular change from the IP log. Currently within JGit we want
to mask only the initial commit, but others could be masked if the
need arises.
Change-Id: I598e08137ddc5913284471ee2aa545f4df685023
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The support for NLS relies on java.util API to load a standard
ResourceBundle and then uses java reflection API to inject localized
strings into public String fields of the corresponding instance
of TranslationBundle.
Locale setting is supported per thread to enable concurrent threads
to use different locales. This is useful when JGit runs in a server
context where (error) messages might need to differ per-request to
suit the user's preference.
Change-Id: Ie0e63a0d7bb74eaad495dbe8248595d8a3a76883
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
When implementing branch read access, we need to prove that the
newly created reference(s) point to objects that the user can see.
There are two ways that an object is reachable:
1) It's reachable from a branch or change the user can see
2) It was uploaded as part of the pack file the user sent us
This change adds additional methods in ReceivePack that will allow a
server to check the above conditions, in order to ensure that a user
is not trying to create a reference that they cannot see, or that a
malicious user isn't attempting to forge the SHA-1 of an object that
they cannot see in order to base a change off of it.
Change-Id: Ieba75b4f0331e06a03417c37f4ae1ebca4fbee5a
If the readAdvertisedRefs() method throws an exception, its already
closed the connection and wrapped the underlying cause inside of a
suitable TransportException object that it is throwing. We shouldn't
catch IOException and rethrow a wrapped copy here, because we'll double
wrap the exception thrown by readAdvertisedRefs. This may obsecure the
root cause of the connection failure from the end-user.
Change-Id: I0ca61560f9888c666323dac8a5582aab25e897ff
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When a user of ReceivePack or UploadPack wants to control what refs
are sent to the client, for instance when some refs should be hidden
from some clients, this interface can be extended to provide a fine
grained control over what refs are sent to the client.
Change-Id: Ie6320b0f8922e1a5e1bad91c016bd476ea094366
The boolean field sentCommand is always true at this point, as it
was assigned just 5 lines above. So we always set the status of
the update command object to AWAITING_REPORT.
Simplify the logic by dropping the ?: operator. I assume this is
older code from an attempt to manage dry-run push support within
the native connection, but in fact dry-run support is done higher
up inside of PushProcess.
Change-Id: I450d491bbbb5afecdbf5444ab7169222e856a3bb
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Windows users by default have core.autocrlf set to true. JGit
does not recognize the flags and thus works as if it is set. In order
to make JGit more compatible with msysgit we set the flag to false
in repositories that JGit creates.
Bug: 301775
Change-Id: I7ea462fe3516e5060b87aa1f7ed63689936830c2
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Doing a keep call with a length of 1 will copy the current entry just
like the previous add was doing, but it avoids doing any validation
on the entry. This is sane because the entry can be assumed to be
already valid, since its originating from the destination index.
Change-Id: I250d902fc98580444af1ba4b8fedceb654541451
Originally: http://thread.gmane.org/gmane.comp.version-control.git/128214/focus=128213
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
A 0 file mode in a DirCacheEntry is not a valid mode. To C git
such a value indicates the record should not be present. We already
were catching this bad state and exceptioning out when writing tree
objects to disk, but we did not fail when writing the dircache back
to disk. This allowed JGit applications to create a dircache file
which C git would not like to read.
Instead of checking the mode during writes, we now check during
mutation. This allows application bugs to be detected sooner and
closer to the cause site. It also allows us to avoid checking most
of the records which we read in from disk, as we can assume these
are formatted correctly.
Some of our unit tests were not setting the FileMode on their test
entry, so they had to be updated to use REGULAR_FILE.
Change-Id: Ie412053c390b737c0ece57b8e063e4355ee32437
Originally: http://thread.gmane.org/gmane.comp.version-control.git/128214/focus=128213
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Adam W. Hawks <awhawks@writeme.com>
A dircache record must not use a path string like "/a" or "a//b"
as this results in a tree entry being written with a zero length
name component in the record. C git does not support an empty name,
and neither does any modern filesystem.
A record also must not have a stage outside of the standard 0-3
value range, as there are only 2 bits of space available in the
on-disk format of the record to store the stage information.
Any other values would be truncated into this space, storing a
different value than the caller expected.
If an application tries to create a DirCache record with either of
these wrong values, we abort with an IllegalArgumentException.
Change-Id: I699de149efdfccd85d8adde07d3efd080e3b49c2
Originally: http://thread.gmane.org/gmane.comp.version-control.git/128214
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Adam W. Hawks <awhawks@writeme.com>
Rather than implementing the file reading logic ourselves, and
wind up leaking the FileInputStream's file descriptor until the
next GC, use IO.readFully(File) which wraps the read loop inside
of a try/finally to ensure the stream is closed before it exits.
Change-Id: I85a3fe87d5eff88fa788962004aebe19d2e91bb4
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Reviewed-by: Roland Grunberg <rgrunber@redhat.com>
ObjectWalk is invoking next() for each record we consider in a tree.
Rather than doing several method calls against the current parser,
and testing if we are at eof() at least twice per next() invocation,
do it only once and inline the logic to move the parser forward.
Change-Id: If5938f5d7b3ca24f500a184c9bd2ef193015414e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The supplied test case comes out of the example tree identified by
Robert de Wilde and Ilari on #git:
$ git ls-tree -rt a54f1a85ebf6a7f53aa60a45a1be33f8b078fb7e
040000 tree bfe058ad536cdb12e127cde63b01472c960ea105 A
040000 tree 4b825dc642 A/A
040000 tree 4b825dc642 A/B
100644 blob abbbfafe3129f85747aba7bfac992af77134c607 B
In this tree, "B" was being skipped because "A/A" as an empty tree
was immediately followed by "A/B", also an empty tree, but the
ObjectWalk broke out too early and never visited "B".
Bug: 286653
Change-Id: I25bcb0bc99d0cbbbdd9c2bd625ad6a691a6d0335
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
During dispose() or reset() we are suppose to be restoring the
ObjectWalk instance back to the original pre-walk state, but we
failed to reset the tree parser. This can lead to confusing state
if the ObjectWalk was reused by the caller, as entries from the
old walk might be reported as part of the new walk.
Change-Id: I6237bae7bfd3794e8b9a92b4dd475559cc72e634
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Instead of including "ObjectId[SHA-1]" in the message, just
us the formatted SHA-1 name of the object by calling name().
Change-Id: I0d1d0e8207f8a3f02188e60242e4e9bf7420e88f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We didn't skip the correct number of bytes when we skipped over an
unrecognized but optional dircache extension. We missed skipping
the 8 byte header that makes up the extension's name and length.
We also didn't include the skipped extension's payload as part of
our index checksum, resuting in a checksum failure when the index
was done reading. So ensure we always scan through a skipped
section and include it in the checksum computation.
Add a test case for a currently unsupported index extension, 'ZZZZ',
to verify we can still read the DirCache object even though we
don't know what 'ZZZZ' is supposed to mean.
Bug: 301287
Change-Id: I4bdde94576fffe826d0782483fd98cab1ea628fa
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If the repository is empty, we have no HEAD branch, which means we
can't test to see if the HEAD is detached and should be advertised
as a .have line.
Change-Id: I6e85f836e7db057cede812d0d6c1aecbd6cbe6c5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The unsetSection method can be used to delete an entire configuration
block, such as a [branch ""] or [remote ""] section in a file.
Change-Id: I93390c9b2187eb1b0d51353518feaed83bed2aad
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Config was confusing the following two variables when writing the
file back to text format:
[my]
empty =
enabled
When parsed, we say that my.empty has 1 value, null, and my.enabled
is an empty string value that in boolean context should be evaluated
as true.
Saving this configuration file back to text format was ignoring the
null value for my.empty, producing a completely different file than
what Config read:
[my]
empty
enabled
Instead handle the writing differently to ensure the original format
is output. New tests cases cover the expected behavior and return
values from accessor methods.
Change-Id: Id37379ce20cb27e3330923cf989444dd9f2bdd96
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We don't want to use the JRE cache when fetching content.
Change-Id: Id76f3e618967c98ed4fbc47a1a2a9e77acbe41ab
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If remote.name.uploadpack or .receivepack is misconfigured and points
to a non-existent command on the remote system, we should receive back
exit status 127. Report this case specially with the command we used
so the user knows what is going.
Bug: 293703
Change-Id: I7504e7b6238d5d8e698d37db7411c4817a039d08
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Annotated tags created with C Git versions before the introduction
of c818566 ([PATCH] Update tags to record who made them, 2005-07-14),
do not have a "tagger" line present in the object header. This line
did not appear in C Git until v0.99.1~9.
Ancient projects such as the Linux kernel contain such tags, for
example Linux 2.6.12 is older than when this feature first appeared
in C Git. Linux v2.6.13-rc4 in late July 2005 is the first kernel
version tag to actually contain a tagger line.
It is therefore acceptable for the header to be missing, and for
the RevTag.getTaggerIdent() method to return null.
Since the Javadoc for getTaggerIdent() already explained that the
identity may be null, we just need to test that this is true when
the header is missing, and allow the ObjectChecker to pass anyway.
Change-Id: I34ba82e0624a0d1a7edcf62ffba72260af6f7e5d
See: http://code.google.com/p/gerrit/issues/detail?id=399
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
* ref-abstract:
Optimize RefAdvertiser performance by avoiding sorting
branch: Add -m option to rename a branch
Replace writeSymref with RefUpdate.link
Rewrite reference handling to be abstract and accurate
Create new RefList and RefMap utility types
Change-Id: If43aacf5aa4013edbd0a6e84d84c4f9e94de5be0
Don't copy and sort the set of references if they are passed through
in a RefMap or a SortedMap using the key's natural sort ordering.
Either map is already in the order we want to present the items
to the client in, so copying and sorting is a waste of local CPU
and memory.
Change-Id: I49ada7c1220e0fc2a163b9752c2b77525d9c82c1
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
By using RefUpdate for symbolic reference creation we can reuse
the logic related to updating the reflog with the event, without
needing to expose something such as the legacy ReflogWriter class
(which we no longer have).
Applications using writeSymref must update their code to use the
new pattern of changing the reference through the updateRef method:
String refName = "refs/heads/master";
RefUpdate u = repository.updateRef(Constants.HEAD);
u.setRefLogMessage("checkout: moving to " + refName, false);
switch (u.link(refName)) {
case NEW:
case FORCED:
case NO_CHANGE:
// A successful update of the reference
break;
default:
// Handle the failure, e.g. for older behavior
throw new IOException(u.getResult());
}
Change-Id: I1093e1ec2970147978a786cfdd0a75d0aebf8010
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This commit actually does three major changes to the way references
are handled within JGit. Unfortunately they were easier to do as
a single massive commit than to break them up into smaller units.
Disambiguate symbolic references:
---------------------------------
Reporting a symbolic reference such as HEAD as though it were
any other normal reference like refs/heads/master causes subtle
programming errors. We have been bitten by this error on several
occasions, as have some downstream applications written by myself.
Instead of reporting HEAD as a reference whose name differs from
its "original name", report it as an actual SymbolicRef object
that the application can test the type and examine the target of.
With this change, Ref is now an abstract type with different
subclasses for the different types.
In the classical example of "HEAD" being a symbolic reference to
branch "refs/heads/master", the Repository.getAllRefs() method
will now return:
Map<String, Ref> all = repository.getAllRefs();
SymbolicRef HEAD = (SymbolicRef) all.get("HEAD");
ObjectIdRef master = (ObjectIdRef) all.get("refs/heads/master");
assertSame(master, HEAD.getTarget());
assertSame(master.getObjectId(), HEAD.getObjectId());
assertEquals("HEAD", HEAD.getName());
assertEquals("refs/heads/master", master.getName());
A nice side-effect of this change is the storage type of the
symbolic reference is no longer ambiguous with the storge type
of the underlying reference it targets. In the above example,
if master was only available in the packed-refs file, then the
following is also true:
assertSame(Ref.Storage.LOOSE, HEAD.getStorage());
assertSame(Ref.Storage.PACKED, master.getStorage());
(Prior to this change we returned the ambiguous storage of
LOOSE_PACKED for HEAD, which was confusing since it wasn't
actually true on disk).
Another nice side-effect of this change is all intermediate
symbolic references are preserved, and are therefore visible
to the application when they walk the target chain. We can
now correctly inspect chains of symbolic references.
As a result of this change the Ref.getOrigName() method has been
removed from the API. Applications should identify a symbolic
reference by testing for isSymbolic() and not by using an arcane
string comparsion between properties.
Abstract the RefDatabase storage:
---------------------------------
RefDatabase is now abstract, similar to ObjectDatabase, and a
new concrete implementation called RefDirectory is used for the
traditional on-disk storage layout. In the future we plan to
support additional implementations, such as a pure in-memory
RefDatabase for unit testing purposes.
Optimize RefDirectory:
----------------------
The implementation of the in-memory reference cache, reading, and
update routines has been completely rewritten. Much of the code
was heavily borrowed or cribbed from the prior implementation,
so copyright notices have been left intact as much as possible.
The RefDirectory cache no longer confuses symbolic references
with normal references. This permits the cache to resolve the
value of a symbolic reference as late as possible, ensuring it
is always current, without needing to maintain reverse pointers.
The cache is now 2 sorted RefLists, rather than 3 HashMaps.
Using sorted lists allows the implementation to reduce the
in-memory footprint when storing many refs. Using specialized
types for the elements allows the code to avoid additional map
lookups for auxiliary stat information.
To improve scan time during getRefs(), the lists are returned via
a copy-on-write contract. Most callers of getRefs() do not modify
the returned collections, so the copy-on-write semantics improves
access on repositories with a large number of packed references.
Iterator traversals of the returned Map<String,Ref> are performed
using a simple merge-join of the two cache lists, ensuring we can
perform the entire traversal in linear time as a function of the
number of references: O(PackedRefs + LooseRefs).
Scans of the loose reference space to update the cache run in
O(LooseRefs log LooseRefs) time, as the directory contents
are sorted before being merged against the in-memory cache.
Since the majority of stable references are kept packed, there
typically are only a handful of reference names to be sorted,
so the sorting cost should not be very high.
Locking is reduced during getRefs() by taking advantage of the
copy-on-write semantics of the improved cache data structure.
This permits concurrent readers to pull back references without
blocking each other. If there is contention updating the cache
during a scan, one or more updates are simply skipped and will
get picked up again in a future scan.
Writing to the $GIT_DIR/packed-refs during reference delete is
now fully atomic. The file is locked, reparsed fresh, and written
back out if a change is necessary. This avoids all race conditions
with concurrent external updates of the packed-refs file.
The RefLogWriter class has been fully folded into RefDirectory
and is therefore deleted. Maintaining the reference's log is
the responsiblity of the database implementation, and not all
implementations will use java.io for access.
Future work still remains to be done to abstract the ReflogReader
class away from local disk IO.
Change-Id: I26b9287c45a4b2d2be35ba2849daa316f5eec85d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
These types can be used by RefDatabase implementations to manage
the collection.
A RefList stores items sorted by their name, and is an immutable
type using copy-on-write semantics to perform modifications to
the collection. Binary search is used to locate an existing item
by name, or to locate the proper insertion position if an item does
not exist.
A RefMap can merge up to 3 RefList collections at once during its
entry iteration, allowing items in the resolved or loose RefList
to override items by the same name in the packed RefList.
The RefMap's goal is O(log N) lookup time, and O(N) iteration time,
which is suitable for returning from a RefDatabase. By relying on
the immutable RefList we might be able to make map construction
nearly constant, making Repository.getAllRefs() an inexpensive
operation if the caches are current. Since modification is not
common, changes require up to O(N + log N) time to copy the internal
list and collapse or expand the list's array. As most changes
are made to the loose collection and not the packed collection,
in practice most changes would require less than the full O(N)
time, due to a significantly smaller N in the loose list.
Almost complete test coverage is included in the corresponding
unit tests. A handful of methods on RefMap are not tested in this
change, as writing the proper test depends on a future refactoring
of how the Ref class represents symbolic reference names.
Change-Id: Ic2095274000336556f719edd75a5c5dd6dd1d857
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
On Windows systems, file system lookup is a slow operation, so
checking each object if it exists during indexing (after receiving
the pack) could take a siginificant time. This patch introduces
CachedObjectDirectory that pre-caches lookup results.
Bug: 300397
Change-Id: I471b93f9bb3ee173eb37cae1d75e9e4eb49985e7
Signed-off-by: Constantine Plotnikov <constantine.plotnikov@gmail.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Not all occurrences of ".git" are replaced by this constant, only
those where it actually refers to the directory with that name, i.e
not the ".git" directory suffix.
Asserts and comment are also excluded from replacement.
Change-Id: I65a9da89aedd53817f2ea3eaab4f9c2bed35d7ee
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
No Eclipse support for this project is provided, because the
Jetty project does not publish a complete P2 repository.
Change-Id: Ic5fe2e79bb216e36920fd4a70ec15dd6ccfd1468
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The dumb HTTP transport needs to download the HEAD ref and
resolve it manually if HEAD does not appear in info/refs.
Its typically for it to not be in the info/refs file.
Change-Id: Ie2a58fdfacfeee530b10edb433b8f98c85568585
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
During fetch over http:// clients now try to take advantage of
the info/refs?service=git-upload-pack URL to determine if the
remote side will support a standard upload-pack command stream.
If so each block of 32 have lines is sent in one POST request,
prefixed by all of the 'want' lines and any previously discovered
common bases as 'have' lines.
During push over http:// clients now try to take advantage of
the info/refs?service=git-receive-pack URL to determine if the
remote side will support a standard receive-pack command stream.
If so, commands are sent along with their pack in a single HTTP
POST request.
Bug: 291002
Change-Id: I8c69b16ac15c442e1a4c3bd60b4ea1a47882b851
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This is a simple HTTP server that provides the minimum server side
support required for dumb (non-git aware) transport clients.
We produce the info/refs and objects/info/packs file on the fly
from the local repository state, but otherwise serve data as raw
files from the on-disk structure.
In the future we could better optimize the FileSender class and the
servlets that use it to take advantage of direct file to network
APIs in more advanced servlet containers like Jetty.
Our glue package borrows the idea of a micro embedded DSL from
Google Guice and uses it to configure a collection of Filters
and HttpServlets, all of which are matched against requests using
regular expressions. If a subgroup exists in the pattern, it is
extracted and used for the path info component of the request.
Change-Id: Ia0f1a425d07d035e344ae54faf8aeb04763e7487
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
By making this class and its methods public, and the actual writing
abstract, we can reuse this code for other formats like writing an
info/refs file for HTTP transports.
Change-Id: Id0e349c30a0f5a8c1527e0e7383b80243819d9c5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If biDirectionalPipe is false UploadPack does not start out with
the advertisement but instead assumes it should read one block of
want/have lines, process that, and write the ACK/NAKs out.
This means it only is doing one read through the input followed by
one write to the output, which fits with the HTTP request processing
model, and any other type of RPC system.
Change-Id: Ia9f7c46ee556f996367180f15d2caa8572cdd59f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If biDirectionalPipe is false ReceivePack does not start out with the
advertisement but instead assumes it should read the command set once,
process that, and write the status report out. This means it only is
doing one read through the input followed by one write to the output,
which fits with the HTTP request processing model, and any other type
of RPC system... assuming that the payload for input can be a very big
entity like the command stream followed by the pack file.
Change-Id: I6f31f6537a3b7498803a8a54e10b0622105718c1
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Later we are going to add support for smart HTTP, which requires us to
buffer at least some of the request created by a client before we ship
it to the server. For many requests, we can fit it completely into a
1 MiB buffer, but if it doesn't we can drop back to using the chunked
transfer encoding to send an unknown stream length.
Rather than recoding the block based memory buffer, we refactor the
local file overflow strategy into a subclass, allowing the HTTP client
code to replace this portion of the logic with its own approach to
start the chunked encoding request.
Change-Id: Iac61ea1017b14e0ad3c4425efc3d75718b71bb8e
Signed-off-by: Shawn O. Pearce <sop@google.com>
The multi_ack_detailed extension breaks out the "ACK %s continue" status
code into "ACK %s common" and "ACK %s ready" states, making it easier to
discover which objects are truely common, and which objects are simply
on a chain the server doesn't care learning about.
Change-Id: Ie8e907424cfbbba84996ca205d49eacf339f9d04
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Under unit tests we want the when and timezone to come from the
MockSystemReader and be stable. We did this for the default
constructor based on the Repository, but failed to do it for the
name,emailAddress variant of the constructor.
Change-Id: I608ac7cf01673729303395e19b379b38fef136b3
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We really mean to omit HEAD here, but botched the difference between
getOrigName and getName on the Ref object. We tested on the wrong
value, picking up the target of the symbolic ref and therefore
included it twice.
Change-Id: If780c65166ccada2e63a4f42bbab752a56b16564
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The C Git documentation stated that the core.worktree config was
not read when the .git directory was found implicitly (from the
working directory).
This was not true, and had not been so for a long time. The
documentation has been updated to document the existing behaviour.
Change-Id: If1e81b6a981b9d70e849f24872f01c110e9bc950
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Deflater can use significant amount of native (i.e. C) heap
space. Failure to promptly release this memory results
in native memory leak in some cases, particularly severe for
VMs with large java max heap size. For example, running
Team->Commit in one of my EGit workspaces results in ~500M
java process size increase without any significant change
to amount of used java heap when JVM is started with -Xmx1024m.
Change-Id: I649679a8df5683ebedd9380d703513d31c625932
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Per CQ 3559 "JGit - Eugene Myers O(ND) difference algorithm" we
have approval to check this into our master branch.
* cq-diff:
Add file content merge algorithm
Add performance tests for MyersDiff
Add javadoc comments, remove unused code, shift comments to correct place
Fixed MyersDiff to be able to handle more than 100k
Fix some warnings regarding unnecessary imports and accessing static methods
Add the "jgit diff" command
Prepare RawText for diff-index and diff-files
Add a test class for Myers' diff algorithm
Add Myers' algorithm to generate diff scripts
Add set to IntList
Conflicts:
org.eclipse.jgit/src/org/eclipse/jgit/diff/RawText.java
Change-Id: Ia8e98d81ba1ab52f84d0258a40e6ef5eece9a5b1
CC: Christian Halstrick <christian.halstrick@sap.com>
Adds the file content merge alorithm and tests for merge to jgit.
The merge algorithm:
- Gets as input parameters the common base, the two new contents
called "ours" and "theirs".
- Computes the Edits from base to ours and from base to theirs with
the help of MyersDiff.
- Iterates over the edits.
- Independent edits from ours or from theirs will just be applied
to the result.
- For conflicting edits we first harmonize the ranges of the edits
so that in the end we have exactly two edits starting and ending
at the same points in the common base. Then we write the two
conclicting contents into the result stream.
Change-Id: I411862393e7bf416b6f33ca55ec5af608ff4663
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
[sp: Fixed up two awkard comments in documentation.]
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The UnionInputStream utility class combines multiple sequential
InputStreams so they appear to the caller as a single stream with
no gaps. This can be used to concentate streams coming from multiple
independent HTTP connections (for example).
The companion unit test covers the class's full functionality.
Change-Id: I0676c7b5e082a5886bf0e8f43f9fd6c46a666228
Signed-off-by: Shawn O. Pearce <sop@google.com>
This exposes the list of known packs, allowing callers to list them
into a context like the objects/info/packs file.
Change-Id: I0b889564bd176836ff5c77ba310c6d229409dcd5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This makes the jgit command line behave like the C Git implementation
in the respect.
These variables are not recognized in the core, though we add support
to do the overrides there. Hence other users of the JGit library, like
the Eclipse plugin and others, will not be affected.
GIT_DIR
The location of the ".git" directory.
GIT_WORK_TREE
The location of the work tree.
GIT_INDEX_FILE
The location of the index file.
GIT_CEILING_DIRECTORIES
A colon (semicolon on Windows) separated list of paths that
which JGit will not cross when looking for the .git directory.
GIT_OBJECT_DIRECTORY
The location of the objects directory under which objects are
stored.
GIT_ALTERNATE_OBJECT_DIRECTORIES
A colon (semicolon on Windows) separated list of object directories
to search for objects.
In addition to these we support the core.worktree config setting when
the git directory is set deliberately instead of being found.
Change-Id: I2b9bceb13c0f66b25e9e3cefd2e01534a286e04c
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
An extra flag when creating a RefUpdate object allows the
caller to destroy the symref and replace it with an object
ref, a.k.a. detached HEAD.
Change-Id: Ia88d48eab1eb4861ebfa39e3be9258c3824a19db
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
A few locations were doing OBJECT_ID_LENGTH * 2 on their own, as
the old STR_LEN constant wasn't visible. Replace them with the
new public constant OBJECT_ID_STRING_LENGTH.
Change-Id: Id39bddb52de8c65bb097de042e9d4ed99598201f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Since Constants.OBJECT_ID_LENGTH is a compile time constant we
can be sure that it will always be inlined. The same goes for the
associated constant STR_LEN which is now refactored to the Constant
class and given a name better suited for wider use.
Change-Id: I03f52131e64edcd0aa74bbbf36e7d42faaf4a698
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
The comment indicates that a well-behaved client should not have
sent an update for a ref that already exists, but this in a block
that corresponds to a create command.
This way SWT based applications don't wind up loading this AWT
based code when using SSH.
Change-Id: I9080f3dd029c2a087e6b687480018997cc5c5d23
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This makes it easier to swap out authenticator implementations and
yet still rely upon being able to configure at least one Authenticator
instance in the JVM and program it with data obtained from outside
of the user interface.
Change-Id: I8c1a0eb8acee1d306f4c3b40a790b7fa0c3abb70
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This new UI bundle contains the org.eclipse.jgit.awtui package,
which was moved out of the org.eclipse.jgit bundle.
org.eclipse.jgit.pgm depends on org.eclipse.jgit.ui, so we need
to update the classpath and make_jgit.sh to include it.
This move takes the awtui classes out of the Maven build, which
means we are no longer able to distribute these classes to our
downstream Maven customers. The entire Maven package structure
needs to be overhauled so that Eclipse bundle matches 1:1 with the
Maven artifact.
Bug: https://bugs.eclipse.org/291124
Change-Id: Ibf1a9968387e3d11fdce54592f710ec4cc7f1ddb
Signed-off-by: Mykola Nikishov <mn@mn.com.ua>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
For historical reasons JGit was creating .git/remotes folder when
initialization an empty repository. However, this folder is never
used and the new Git repository format doesn't contain them.
Removed the unnecessary remotes folder creation, it was removed from
C Git in 75c384efb52d ("Do not create $GIT_DIR/remotes/ directory
anymore") by Junio C Hamano on Dec 19 2006. C Git hasn't made it
for almost 3 years.
Change-Id: I5473f3676a3642afdf9aaac44b7e6319c6e170b9
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
According the javadoc, and implied by the name of the class, NB
is about network byte order. The purpose of moving the IO only,
and non-byte order related functions to another class is to
make it easier for new contributors to understand that they
can use these functions in general and it's also makes it easier
to understand where to put new IO related utility functions
Change-Id: I4a9f6b39d5564bc8a694b366e7ff3cc758c5181b
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
As discussed on the egit-dev mailing list, we prefer not to have
trailing whitespace in our source code. Correct all currently
offending lines by trimming them.
Change-Id: I002b1d1980071084c0bc53242c8f5900970e6845
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This change only fixes warnings of the eclipse build regarding missing javadocs.
Some comments where just missing, so they have been added. Other comments where
at the wrong (from eclipse point of view) place, so eclipse was complaining.
Also two method which existed for debugging purposes have been removed to get
rid of Eclipse warngins about unused code.
MyersDiff was crashing with ArrayIndexOutOfBoundsException when
diffing huge files. This was because the snake data (begin and
end position while processing chunks of the text) was sequeezed
into a single int. But with longer texts to diff these values
exceeded the range of java int values. It is fixed by squeezing
the two ints into a long and by adding the LongList helper
class.
Change-Id: Iacb9082e1b076e994d1486aa8e512342ad7432b3
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
When reading commits the system default encoding was used if no
encoding was specified in the commit. The patch modifies the test
to add a check that commit message was encoded correctly (the
test fails on old implementation if system encoding is not UTF-8)
and fixes Commit.decode() method to use UTF-8 if encoding is not
specified in the commit object.
Change-Id: I27101da3c2eb6edd0c4a9e4c0523e48b286e3cd5
Signed-off-by: Constantine Plotnikov <constantine.plotnikov@gmail.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This commit contains fixes provided by Christian Halstrick.
Bug: 291083
Eclipse-CQ: 3559
Change-Id: If8d187e70a51f31040f99098489e513791dc6415
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Myers' algorithm is the standard way to generate diff scripts in
an efficient manner (especially memory-wise).
The source contains extensive documentation about the principal
ideas of the algorithm.
Bug: 291083
Eclipse-CQ: 3559
Change-Id: Id873e7c2c204692cb4a1a05d24b733154b9ed49c
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some applications may wish to modify an int list.
Bug: 291083
Eclipse-CQ: 3559
Change-Id: Iea871443ec661230aec92397229f1eda6c74216f
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This is the last chunk of code in jgit-core which references the awtui
package. Moving it to the only consumer in jgit-pgm allows us to move
the awtui package over to the jgit-awtui module.
Change-Id: I2fd81be2076117b2f2c5f8ed45de7f29272af6cf
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This removes one of the few remaining dependencies on AWTAuthenticator
from the core library. For the most part the interface is identical to
the prior main method. The jgit-pgm Main class already sets up the
HTTP proxy and authenticator for us, so we don't need to do that in
our new run method.
Change-Id: Ia2cce34d34c97b88214a8bd8f2cc542845e19032
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Honor the configuration parameter core.logAllRefUpdates when writing
reflogs. Instead of writing reflog entries always only write
reflogs if this parameter is set to true or if the corresponding
file in the <git-dir>/logs directory already exists. In other words:
if you are updating a ref and this parameter is set to false and
there is no file corresponding to your ref in the <git-dir>/logs
folder then no reflog will be written.
This is a fix for the issue http://code.google.com/p/egit/issues/detail?id=4
Change-Id: I908e4c77e3630dc3223b2d2a47cb4534dbe4ed42
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Recently Eclipse started complaining about undocumented protected
members here, so lets document them to make the errors go away.
Change-Id: Ia3bc9fa69854953db5dc9a982821a2f3abe8d05a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This allows SwingGraphPanes to be reused by simply clearing and
re-filling. Requires RevObjectList initialization to not call clear()
from its constructor, because this will lead PlotCommitList.clear()
to be called before all variables have been initialized.
Change-Id: I14a07124441b58cd88c67da088ba52ef9c30b043
Signed-off-by: Jonas Fonseca <fonseca@diku.dk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When throwing MissingBundlePrerequisiteException we
also include the short description, if available, of
each missing object.
This is the fix for the following issue:
http://code.google.com/p/egit/issues/detail?id=25
Change-Id: I5d45aec7873af76a12170d9a500626a7264f2c42
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Per CQ 3448 this is the initial contribution of the JGit project
to eclipse.org. It is derived from the historical JGit repository
at commit 3a2dd9921c8a08740a9e02c421469e5b1a9e47cb.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce