Commit Graph

155 Commits

Author SHA1 Message Date
Shawn O. Pearce 60c5939b23 Rename getOldName,getNewName to getOldPath,getNewPath
TreeWalk calls this value "path", while "name" is the stuff after the
last slash.  FileHeader should do the same thing to be consistent.
Rename getOldName to getOldPath and getNewName to getNewPath.

Bug: 318526
Change-Id: Ib2e372ad4426402d37939b48d8f233154cc637da
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-08-04 11:00:07 -07:00
Shawn O. Pearce 7514a6dbdc Merge branch 'js/diff'
* js/diff:
  Fixed bug in scoring mechanism for rename detection
2010-08-04 10:59:35 -07:00
Jeff Schumacher e64cb03065 Fixed bug in scoring mechanism for rename detection
A bug in rename detection would cause file scores to be wrong. The
bug was due to the way rename detection would judge the similarity
between files. If file A has three lines containing 'foo', and file
B has 5 lines containing 'foo', the rename detection phase should
record that A and B have three lines in common (the minimum of the
number of times that line appears in both files). Instead, it would
choose the the number of times the line appeared in the destination
file, in this case file B. I fixed the bug by having the
SimilarityIndex instead choose the minimum number, as it should. I
also added a test case to verify that the bug had been fixed.

Change-Id: Ic75272a2d6e512a361f88eec91e1b8a7c2298d6b
2010-08-04 10:56:19 -07:00
Jens Baumgart 3ba1c7c068 Add gitignore support to IndexDiff and use TreeWalk
IndexDiff was re-implemented and now uses TreeWalk instead
of GitIndex. Additionally, gitignore support and retrieval of
untracked files was added.

Change-Id: Ie6a8e04833c61d44c668c906b161202b200bb509
Signed-off-by: Jens Baumgart <jens.baumgart@sap.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
2010-08-04 10:03:20 -05:00
Stefan Lay d9ceaa2f1e Fixed FileTreeIteratorWithTimeControl
getEntryLastModified returned wrong values for
timestamps on boundaries.

Change-Id: I0457ff030946fda7ecc70671bc65db02a8c206ee
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
2010-07-30 17:35:29 +02:00
Shawn O. Pearce 7b8e8df799 Remove use of SortedSet.floor()
Its only available in Java 6, and JGit still supports Java 5.

Change-Id: I6c8fac06106ffe20ac8b4b595f34e027f2d29cc2
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-28 14:40:09 -07:00
Christian Halstrick 4ed2f94013 Add tests for racy git / Index state encoding
In order to test racy git situations we have to be able to control
the last-modification timestamps of the filesystem. Since we already
access the modification timestamps of files through an abstraction
(the WorkingTreeIterator) I add a new implementation of this iterator
which allows to map timestamp-ranges to single constant timestamps.
For users of this iterator it looks like all files in that range
have been modified at exactly the same time.
With the help of this iterator a test has been writting which
checkes for racy git handling (smudging, unsmudging, dirty-detection).

Additionally add a method to RepositoryTestCase which encodes the
current index state in one String. This should include info about
pathes, file/index modtime, smudgeState, clean-state. Make
sure timestamps are presented in a way that it is easy to
write assertions against this strings (no concrete milliseconds
but t0,t1,...).

These two topics depend circulary on each other: thats why they have
been squashed in one commit.

Change-Id: I115c3f2f20fca9b481830bdc6b9d1ade2c3abdcf
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
2010-07-28 12:16:23 -07:00
Christian Halstrick 74d279fbf0 Teach NameConflictTreeWalk to report DF conflicts
Add a method isDirectoryFileConflict() to NameConflictTreeWalk which
tells whether the current path is part of a directory/file conflict.

Change-Id: Iffcc7090aaec743dd6f3fd1a333cac96c587ae5d
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-28 17:26:31 +02:00
Jeff Schumacher 396fe6da45 Break dissimilar file pairs during diff
File pairs that are very dissimilar during a diff were not being
broken apart into their constituent ADD/DELETE pairs. The leads to
sub-optimal rename detection. Take, for example, this situation:

A file exists at src/a.txt containing "foo". A user renames src/a.txt
to src/b.txt, then adds a new src/a.txt containing "bar".

Even though the old a.txt and the new b.txt are identical, the
rename detection algorithm would not detect it as a rename since
it was already paired in a MODIFY. I added code to split all
MODIFYs below a certain score into their constituent ADD/DELETE
pairs. This allows situations like the one I described above to be
more correctly handled.

Change-Id: I22c04b70581f206bbc68c4cd1ee87a1f663b418e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-27 18:13:32 -07:00
Christian Halstrick f56a459966 Add methods which write MERGE_HEAD and MERGE_MSG
Add methods to the Repository class which write into MERGE_HEAD
and MERGE_MSG files. Since we have the read methods in the same
class this seems to be the right place.

Change-Id: I5dd65306ceb06e008fcc71b37ca3a649632ba462
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-27 11:48:23 -07:00
Jens Baumgart db82b8d7eb Fix concurrent read / write issue in LockFile on Windows
LockFile.commit fails if another thread concurrently reads
the base file. The problem is fixed by retrying the rename
operation if it fails.

Change-Id: I6bb76ea7f2e6e90e3ddc45f9dd4d69bd1b6fa1eb
Bug: 308506
Signed-off-by: Jens Baumgart <jens.baumgart@sap.com>
2010-07-27 10:00:47 -07:00
Robin Stocker a00377a7e2 Fix Javadoc warnings
There were some broken links, incorrect uses of @value, an invalid
tag and an outdated comment.

Change-Id: I22886bcc869a4b62bd606ebed40669f7b4723664
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-27 09:40:01 -07:00
Shawn O. Pearce fa9b225e06 Merge branch 'delta'
* delta: (103 commits)
  Discard the uncompressed delta as soon as its compressed
  Honor pack.windowlimit to cap memory usage during packing
  Honor pack.threads and perform delta search in parallel
  Cache small deltas during packing
  Implement delta generation during packing
  debug-show-packdelta:  Dump a pack delta to the console
  Initial pack format delta generator
  Add debugging toString() method to ObjectToPack
  Make ObjectToPack clearReuseAsIs signal available to subclasses
  Correctly classify the compressing objects phase
  Refactor ObjectToPack's delta depth setting
  Configure core.bigFileThreshold into PackWriter
  Add doNotDelta flag to ObjectToPack
  Add more configuration options to PackWriter
  Save object path hash codes during packing
  Add path hash code to ObjectWalk
  Add getObjectSize to ObjectReader
  Allow TemporaryBuffer.Heap to allocate smaller than 8 KiB
  Define a constant for 127 in DeltaEncoder
  Cap delta copy instructions at 64k
  ...

Conflicts:
	org.eclipse.jgit.pgm/src/org/eclipse/jgit/pgm/Diff.java
	org.eclipse.jgit/resources/org/eclipse/jgit/JGitText.properties
	org.eclipse.jgit/src/org/eclipse/jgit/JGitText.java
	org.eclipse.jgit/src/org/eclipse/jgit/revwalk/RewriteTreeFilter.java

Change-Id: I7c7a05e443a48d32c836173a409ee7d340c70796
2010-07-22 14:56:34 -07:00
Stefan Lay 88957f6c5a Allow for filepattern "." in AddCommand
Enable adding on repository root level.

Change-Id: I415b10dc74cc9435578424d9f106c972fd703055
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
2010-07-22 14:27:35 +02:00
Stefan Lay aa86cfc339 Do not add ignored files in Add command
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
2010-07-22 11:26:04 +02:00
Shawn O. Pearce 09910ffa32 Move ignore node handling into WorkingTreeIterator
The working tree iterator has perfect knowledge of the path structure
as well as immediate information about whether or not an ignore file
even exists at this level.  We can exploit that to simplify the
logic and running time for testing ignored file status by pushing
all of the checks down into the iterator itself.

Change-Id: I22ff534853e8c5672cc5c2d9444aeb14e294070e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Charley Wang <chwang@redhat.com>
CC: Chris Aniszczyk <caniszczyk@gmail.com>
CC: Stefan Lay <stefan.lay@sap.com>
CC: Matthias Sohn <matthias.sohn@sap.com>
2010-07-21 10:34:08 -07:00
Shawn O. Pearce 395d236058 Fix NPE in RenameDetector
If we have two adds of the same object but no deletes the detector
threw an NPE because the entry that came back from the deleted map
was null (no matching objects).  In this case we need to put the
adds all back onto the list of left over additions since they did
not match a delete.

Change-Id: Ie68fbe7426b4dc0cb571a08911c7adbffff755d5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Jeffrey Schumacher" <jeffschu@google.com>
2010-07-20 07:52:35 -07:00
Shawn O. Pearce 6e155d5f41 Merge branch 'js/rename'
* js/rename:
  Implemented file path based tie breaking to exact rename detection
  Added more test cases for RenameDetector
  Added very small optimization to exact rename detection
  Fixed Misleading Javadoc
  Added file path similarity to scoring metric in rename detection
  Fixed potential div by zero bug
  Added file size based rename detection optimization
  Create FileHeader from DiffEntry
  log: Implement --follow
  Cache the diff configuration section
  log: Add whitespace ignore options
  Format submodule links during differences
  Redo DiffFormatter API to be easier to use
  log, diff: Add rename detection support
  Implement similarity based rename detection
  Added a preliminary version of rename detection
  Refactored code out of FileHeader to facilitate rename detection
2010-07-16 10:22:15 -07:00
Jeff Schumacher 31311cacfd Implemented file path based tie breaking to exact rename detection
During the exact rename detection phase in RenameDetector, ties were
resolved on a first-found basis. I added support for file path based
tie breaking during that phase. Basically, there are four situations
that have to be handled:

One add matching one delete:
	In this simple case, we pair them as a rename.

One add matching many deletes:
	Find the delete whos path matches the add the closest, and
	pair them as a rename.

Many adds matching one delete:
	Similar to the above case, we find the add that matches the
	delete the closest, and pair them as a rename. The other adds
	are marked as copies of the delete.

Many adds matching many deletes:
	Build a scoring matrix similar to the one used for content-
	based matching, scoring instead by file path. Some of the
	utility functions in SimilarityRenameDetector are used in
	this case, as we use the same encoding scheme. Once the
	matrix is built, scan it for the best matches, marking them
	as renames. The rest are marked as copies.

I don't particularly like the idea of using utility functions right
out of SimilarityRenameDetector, but it works for the moment. A later
commit will likely refactor this into a common utility class, as well
as bringing exact rename detection out of RenameDetector and into a
separate class, much like SimilarityRenameDetector.

Change-Id: I1fb08390aebdcbf20d049aecf402a36506e55611
2010-07-16 09:56:42 -07:00
Shawn Pearce ff59ce4bff Merge "Remove an unused File reference in test code" 2010-07-15 19:01:25 -04:00
Robin Rosenberg 9d589c88f7 Remove an unused File reference in test code
Change-Id: Ib0d6c36811df719a53c66e9fa7460b89b2faf98b
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
2010-07-16 00:35:47 +02:00
Shawn Pearce 19473b1dbc Merge "Handle the tilde notation (~user) of git url" 2010-07-15 17:29:21 -04:00
Robin Rosenberg 845714158a Handle the tilde notation (~user) of git url
When the path is prefixed with ~ the URI parser thought about this
as /~. Strip the / if the next character is the tilde.

Bug: 307017
Change-Id: I58203e5617956b46d83e8987d1f8042beddffac3
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
2010-07-15 01:16:09 +02:00
Stefan Lay 233e0130b5 Git Porcelain API: Add Command
The new Add command adds files to the Git Index. 
It  uses the DirCache to access the git index. It 
works also in case of an existing conflict. 

Fileglobs (e.g. *.c) are not yet supported. 

The new Add command does add ignored files because
there is no gitignore support in jgit yet.

Bug: 318440
Change-Id: If16fdd4443e46b27361c2a18ed8f51668af5d9ff
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
2010-07-14 11:24:58 +00:00
Christian Halstrick 33160cd2da Fix ReadTreeTest
After refactoring ReadTreeTest the tests failed for filesystems
with coarse modification time granularity. This is fixed by
explicitly telling the repo to reread the index after we build
a new index.

Additionally the test testDirectoryFileSimple was simplified
by using buildTree() instead of misusing GitIndex to construct
trees.

Change-Id: I20d2f097491e4cc8c657a696beabc7026b485017
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2010-07-13 15:53:14 +02:00
Charley Wang b878cdcf6b Add compatibility with gitignore specifications
This patch adds ignore compatibility to jgit. It encompasses
exclude files as well as .gitignore. Uses TreeWalk and
FileTreeIterator to find nodes and parses .gitignore
files when required. The patch includes a simple cache that
can be used to save results and avoid excessive gitignore
parsing.

CQ: 4302
Bug: 303925
Change-Id: Iebd7e5bb534accca4bf00d25bbc1f561d7cad11b
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2010-07-13 00:34:15 +02:00
Jeff Schumacher f666cc755b Added more test cases for RenameDetector
I added test cases to cover the majority of the code. It's not 100%
coverage yet, but the remaining bits are small.

Change-Id: Ib534c8e94b13358b8b22cf54e2ff84132bae6d14
2010-07-12 14:48:31 -07:00
Jeff Schumacher 9a48de86d8 Added file path similarity to scoring metric in rename detection
The scoring method was not taking into account the similarity of
the file paths and file names. I changed the metric so that it is 99%
based on content (which used to be 100% of the old metric), and 1%
based on path similarity. Of that 1%, half (.5% of the total final
score) is based on the actual file names (e.g. "foo.java"), and half
on the directory (e.g. "src/com/foo/bar/").

Change-Id: I94f0c23bf6413c491b10d5625f6ad7d2ecfb4def
2010-07-12 12:52:05 -07:00
Shawn O. Pearce 8612c0ace1 Initial pack format delta generator
DeltaIndex is a simple pack style delta generator.  The function works
by creating a compact index of a source buffer's blocks, and then
walking a sliding window along a desired result buffer, searching for
the window in the index.  When a match is found, the window is
stretched to the longest possible length that is common with the
source buffer, and a copy instruction is created.

Rabin's polynomial hash function is used to compute the hash for a
block, permitting efficient sliding of the window in single byte
increments.  The update function to slide one byte originated from
David Mazieres' work in LBFS, and our implementation of the update
step was certainly inspired by the initial work Geert Bosch proposed
for C Git in http://marc.info/?l=git&m=114565424620771&w=2.

To ensure the encoder runs in linear time with respect to the size of
the two input buffers (source and result), the maximum number of
blocks that can share the same position in the index's hashtable is
capped at a constant number.  This prevents bad inputs from causing
the encoder to run in quadratic time, but comes with a penalty of
creating a longer delta due to fewer considered copy positions.

Strange hackery is used to cap the amount of memory used by the index
to be no more than 12 bytes for every 16 bytes of source buffer, no
matter what the JVM per-object overhead is.  This permits an index to
always be no larger than 1.75x the source buffer length, which is an
important feature to support large windows of candidates to match
against while packing.  Here the strange hackery is nothing more than
a manually managed chained hashtable, where pointers are array indexes
into storage arrays rather than object references.

Computation of the hash function for a single fixed sized block is
done through an unrolled loop, where the first 4 iterations have been
manually reduced down to eliminate unnecessary instructions.  The
pattern is derived from ObjectId.equals(byte[], int, byte[], int),
where we have unrolled the loop required to compare two 20 byte
arrays.  Hours of testing with the Sun 1.6 JRE concluded that the
non-obvious "foo[idx + 1]" style of reference is faster than
"foo[idx++]", and so that is what we use here during hashing.

Change-Id: If9fb2a1524361bc701405920560d8ae752221768
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-09 19:10:55 -07:00
Shawn O. Pearce 97311cd3e0 Allow TemporaryBuffer.Heap to allocate smaller than 8 KiB
If the heap limit was set to something smaller than 8 KiB, we were
still allocating the full 8 KiB block size, and accepting up to
the amount we allocated by.  Instead actually put a hard cap on
the limit.

Change-Id: Id1da26fde2102e76510b1da4ede8493928a981cc
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-09 10:37:47 -07:00
Jeff Schumacher a8b29afd82 Create FileHeader from DiffEntry
Added support for converting DiffEntrys to FileHeaders. FileHeaders
are DiffEntrys with a buffer containing the diff output as well as
a list of HunkHeaders. The HunkHeaders contain EditLists. The
createFileHeader(DiffEntry) method in DiffFormatter performs a Myers
Diff on the files refered to by the DiffEntry, then puts the returned
EditList into a single HunkHeader, which is then put into the
FileHeader to be returned. It also generates the appropriate diff
header an puts it into the FileHeader's buffer. The rest of the diff
output, which would normally be parsed to generate the HunkHeaders,
is not generated. In fact, the purpose of this method is to avoid
the costly diff output generation and parsing normally required to
create a FileHeader.

Change-Id: I7d8b18c0f6c85e3d02ad58995d3d231e69af5887
2010-07-08 16:58:55 -07:00
Christian Halstrick 4be88168b6 Allow ReadTreeTest to test arbitrary Checkouts
ReadTreeTest was hardcoded to test WorkDirCheckout. Since we want
alternative checkout implementations (especially DirCacheCheckout)
this class has been refactored so that the tests can be reused
to test other implementations

The following changes have been done:
- abstract methods for checkout and prescanTwoTrees have been
  introduced. Parameters are only the two trees. As index we
  will implicitly use the current index of the repo.
- whenever tests needed a manipulated index before checkout
  and prescanTwoTrees it was ensured that the correct index was
  persisted (before we could use not-persisted instantiations of GitIndex
  passed as parameters to checkout, prescanTwoTrees
- abstract methods for getting updated, conflicting, removed entries
  resulting from the last checkout, prescanTwoTrees have been introduced
- an implementation for all these abstract methods using WorkDirCheckout
  has been added
- method to assert a certain state of the index and the working tree has
  been added

Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Change-Id: 	Icf177cf8043487169a32ddd72b6f8f9246a433f7
2010-07-09 01:38:49 +02:00
Shawn O. Pearce 384a19eee0 Deprecate all of the older Tree related code
We want to get rid of these APIs, because they don't perform as well
as DirCache/TreeWalk, or don't offer nearly as many features.

Bug: 319145
Change-Id: I2b28f9cddc36482e1ad42d53e86e9d6461ba3bfc
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-07 09:15:02 -07:00
Shawn O. Pearce ab3c68c512 amend commit: Support large loose objects as streams
We need to validate the stream state after the InflaterInputStream
thinks the stream is done.  Git expects a higher level of service from
the Inflater than the InflaterInputStream usually gives, we need to
ensure the embedded CRC is valid, and that there isn't trailing
garbage at the end of the file.

Change-Id: I1c9642a82dbd76b69e607dceccf8b85dc869a3c1
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-06 19:41:01 -07:00
Shawn O. Pearce 5be90be996 Redo DiffFormatter API to be easier to use
Passing around the OutputStream and the Repository is crazy.  Instead
put the stream in the constructor, since this formatter exists only to
output to the stream, and put the repository as a member variable that
can be optionally set.

Change-Id: I2bad012fee7f40dc1346700ebd19f1e048982878
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-03 16:58:37 -07:00
Shawn O. Pearce 978535b090 Implement similarity based rename detection
Content similarity based rename detection is performed only after
a linear time detection is performed using exact content match on
the ObjectIds.  Any names which were paired up during that exact
match phase are excluded from the inexact similarity based rename,
which reduces the space that must be considered.

During rename detection two entries cannot be marked as a rename
if they are different types of files.  This prevents a symlink from
being renamed to a regular file, even if their blob content appears
to be similar, or is identical.

Efficiently comparing two files is performed by building up two
hash indexes and hashing lines or short blocks from each file,
counting the number of bytes that each line or block represents.

Instead of using a standard java.util.HashMap, we use a custom
open hashing scheme similiar to what we use in ObjecIdSubclassMap.
This permits us to have a very light-weight hash, with very little
memory overhead per cell stored.

As we only need two ints per record in the map (line/block key and
number of bytes), we collapse them into a single long inside of
a long array, making very efficient use of available memory when
we create the index table.  We only need object headers for the
index structure itself, and the index table, but not per-cell.
This offers a massive space savings over using java.util.HashMap.

The score calculation is done by approximating how many bytes are
the same between the two inputs (which for a delta would be how much
is copied from the base into the result).  The score is derived by
dividing the approximate number of bytes in common into the length
of the larger of the two input files.

Right now the SimilarityIndex table should average about 1/2 full,
which means we waste about 50% of our memory on empty entries
after we are done indexing a file and sort the table's contents.
If memory becomes an issue we could discard the table and copy all
records over to a new array that is properly sized.

Building the index requires O(M + N log N) time, where M is the
size of the input file in bytes, and N is the number of unique
lines/blocks in the file.  The N log N time constraint comes
from the sort of the index table that is necessary to perform
linear time matching against another SimilarityIndex created for
a different file.

To actually perform the rename detection, a SxD matrix is created,
placing the sources (aka deletions) along one dimension and the
destinations (aka additions) along the other.  A simple O(S x D)
loop examines every cell in this matrix.

A SimilarityIndex is built along the row and reused for each
column compare along that row, avoiding the costly index rebuild
at the row level.  A future improvement would be to load a smaller
square matrix into SimilarityIndexes and process everything in that
sub-matrix before discarding the column dimension and moving down
to the next sub-matrix block along that same grid of rows.

An optional ProgressMonitor is permitted to be passed in, allowing
applications to see the progress of the detector as it works through
the matrix cells.  This provides some indication of current status
for very long running renames.

The default line/block hash function used by the SimilarityIndex
may not be optimal, and may produce too many collisions.  It is
borrowed from RawText's hash, which is used to quickly skip out of
a longer equality test if two lines have different hash functions.
We may need to refine this hash in the future, in order to minimize
the number of collisions we get on common source files.

Based on a handful of test commits in JGit (especially my own
recent rename repository refactoring series), this rename detector
produces output that is very close to C Git.  The content similarity
scores are sometimes off by 1%, which is most probably caused by
our SimilarityIndex type using a different hash function than C
Git uses when it computes the delta size between any two objects
in the rename matrix.

Bug: 318504
Change-Id: I11dff969e8a2e4cf252636d857d2113053bdd9dc
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-03 16:32:03 -07:00
Shawn O. Pearce 08d349a27b amend commit: Refactor repository construction to builder class
During code review, Alex raised a few comments about commit
532421d989 ("Refactor repository construction to builder class").
Due to the size of the related series we aren't going to go back
and rebase in something this minor, so resolve them as a follow-up
commit instead.

Change-Id: Ied52f7a8f7252743353c58d20bfc3ec498933e00
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-03 10:54:30 -07:00
Shawn O. Pearce 113577617b Use core.streamFileThreshold to set our streaming limit
We default this to 1 MiB for now, but we allow users to modify
it through the Repository's configuration file to be a different
value.  A new repository listener is used to identify when the
setting has been updated and trigger a reconfiguration of any
active ObjectReaders.

To prevent a horrible explosion we cap core.streamFileThreshold
at no more than 1/4 of the maximum JVM heap size.  We do this
because we need at least 2 byte arrays equal in size to the
stream threshold for the worst case delta inflation scenario,
and our host application probably also needs some amount of the
heap for their working set size.

Change-Id: I103b3a541dc970bbf1a6d92917a12c5a1ee34d6c
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-02 12:41:39 -07:00
Shawn O. Pearce ad68553be4 Support large delta packed objects as streams
Very large delta instruction streams, or deltas which use very large
base objects, are now streamed through as large objects rather than
being inflated into a byte array.

This isn't the most efficient way to access delta encoded content, as
we may need to rewind and reprocess the base object when there was a
block moved within the file, but it will at least prevent the JVM from
having its heap explode.

When streaming a delta we have an inflater open for each level in the
delta chain, to inflate the instruction set of the delta, as well as
an inflater for the base level object.  The base object is buffered,
as is the top level delta requested by the application, but we do not
buffer the intermediate delta streams.  This keeps memory usage lower,
so its closer to 1024 bytes per level in the chain, without having an
adverse impact on raw throughput as the top-level buffer gets pushed
down to the lowest stream that has the next region.

Delta instructions transparently collapse here, if the top level does
not copy a region from its base, the base won't materialize that part
from its own base, etc.  This allows us to avoid copying around a lot
of segments which have been deleted from the final version.

Change-Id: I724d45245cebb4bad2deeae7b896fc55b2dd49b3
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-02 02:19:12 -07:00
Shawn O. Pearce ded8f6c721 Support large whole packed objects as streams
Similar to the loose object support, whole packed objects can
now be streamed back to the caller.  The streaming is less
efficient as we copy the data from the cached window array
into the InflaterInputStream's internal buffer, then inflate
it there before returning to the application.

Like with unpacked objects, there is plenty of room for some
optimization, especially for the copyTo method, where we don't
necessarily need so much buffering to exist.

Change-Id: Ie23be81289e37e24b91d17b0891e47b9da988008
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-01 19:34:21 -07:00
Shawn O. Pearce 13e0218a25 Replace PackedObjectLoader with ObjectLoader.SmallObject
The class is identical, but ObjectLoader.SmallObject is part of our
public API for storage implementations to build on top of.

Change-Id: I381a3953b14870b6d3d74a9c295769ace78869dc
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-01 18:27:51 -07:00
Shawn O. Pearce fa23482ca7 Support large loose objects as streams
Big loose objects can now be streamed if they are over the large
object size threshold.  This prevents the JVM heap from exploding
with a very large byte array to hold the slurped file, and then
again with its uncompressed copy.

We may have slightly slowed down the simple case for small
loose objects, as the loader no longer slurps the entire thing
and decompresses in memory.  To try and keep good performance
for the very common small objects that are below 8 KiB in size,
buffers are set to 8 KiB, causing the reader to slurp most of the
file anyway.  However the data has to be copied at least once,
from the BufferedInputStream into the InflaterInputStream.

New unit tests are supplied to get nearly 100% code coverage on the
unpacked code paths, for both standard and pack style loose objects.
We tested a fair chunk of the code elsewhere, but these new tests
are better isolated to the specific branches in the code path.

Change-Id: I87b764ab1b84225e9b5619a2a55fd8eaa640e1fe
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-07-01 18:26:17 -07:00
Jeff Schumacher cb8e1e6014 Added a preliminary version of rename detection
JGit does not currently do rename detection during diffs. I added
a class that, given a TreeWalk to iterate over, can output a list
of DiffEntry's for that TreeWalk, taking into account renames. This
class only detects renames by SHA1's. More complex rename detection,
along the lines of what C Git does will be added later.

Change-Id: I93606ce15da70df6660651ec322ea50718dd7c04
2010-07-01 17:33:53 -07:00
Shawn O. Pearce a1d5f5b6b5 Move DirCache factory methods to Repository
Instead of creating the DirCache from a static factory method, use
an instance method on Repository, permitting the implementation to
override the method with a completely different type of DirCache
reading and writing.  This would better support a repository in the
cloud strategy, or even just an in-memory unit test environment.

Change-Id: I6399894b12d6480c4b3ac84d10775dfd1b8d13e7
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-06-30 10:39:00 -07:00
Shawn O. Pearce cb9d8285ba Create NoWorkTreeException for bare repositories
Using a custom exception type makes it easire for an application
developer to understand why an exception was thrown out of a method
we declare.  To remain compatiable with existing callers, we still
extend off IllegalStateException.

Change-Id: Ideeef2399b11ca460a2dbb3cd80eb76aa0a025ba
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-06-30 09:48:36 -07:00
Shawn O. Pearce 121d009b9b Use ObjectReader in RevWalk, TreeWalk
We don't actually need a Repository object here, just an ObjectReader
that can load content for us.  So change the API to depend on that.

However, this breaks the asCommit and asTag legacy translation methods
on RevCommit and RevTag, so we still have to keep the Repository
inside of RevWalk for those two types.  Hopefully we can drop those in
the future, and then drop the Repository off the RevWalk.

Change-Id: Iba983e48b663790061c43ae9ffbb77dfe6f4818e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-06-28 18:47:29 -07:00
Shawn Pearce 3fd4918852 Merge changes Ie56301aa,Ic2f79e85
* changes:
  Added further support for whitespace ignoring during diff
  Added support for whitespace ignoring
2010-06-28 20:27:04 -04:00
Jeff Schumacher 9869ef2592 Added further support for whitespace ignoring during diff
Added code to support ignoring leading, trailing, and changed
whitespace when performing a diff operation. I also added command
line options to Diff to enable the various whitespace ignoring
methods. These match the flags for git diff.

Change-Id: Ie56301aafad59ee3f0fe5de62719f5023cd702c8
2010-06-28 17:25:19 -07:00
Shawn O. Pearce acb7be2c5a Refactor Repository.openObject to be Repository.open
We drop the "Object" suffix, because its pretty clear here that
we want to open an object, given that we pass in AnyObjectId as
the main parameter.  We also fix the calling convention to throw
a MissingObjectException or IncorrectObjectTypeException, so that
callers don't have to do this error checking themselves.

Change-Id: I72c43353cea8372278b032f5086d52082c1eee39
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-06-28 11:54:58 -07:00
Shawn O. Pearce 6b62e53b60 Move PackWriter progress monitors onto the operations
Rather than taking the ProgressMonitor objects in our constructor and
carrying them around as instance fields, take them as arguments to the
actual time consuming operations we need to run.

Change-Id: I2b230d07e277de029b1061c807e67de5428cc1c4
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2010-06-28 11:47:28 -07:00