Commit Graph

3062 Commits

Author SHA1 Message Date
Terry Parker 2524157d3d Limit the range of commits for which bitmaps are created.
A bitmap index contains bitmaps for a set of commits in a pack file.
Creating a bitmap for every commit is too expensive, so heuristics
select the most "important" commits. The most recent commits are the
most valuable. To clone a repository only those for the branch tips are
needed. When fetching, only commits since the last fetch are needed.

The commit selection heuristics generally work, but for some
repositories the number of selected commits is prohibitively high. One
example is the MSM 3.10 Linux kernel. With over 1 million commits on
2820 branches, the current heuristics resulted in +36k selected commits.
Each uncompressed bitmap for that repository is ~413k, making it
difficult to complete a GC operation in available memory.

The benefit of creating bitmaps over the entire history of a repository
like the MSM 3.10 Linux kernel isn't clear. For that repository, most
history for the last year appears to be in the last 100k commits.
Limiting bitmap commit selection to just those commits reduces the count
of selected commits from ~36k to ~10.5k. Dropping bitmaps for older
commits does not affect object counting times for clones or for fetches
on clients that are reasonably up-to-date.

This patch defines a new "bitmapCommitRange" PackConfig parameter to
limit the commit selection process when building bitmaps. The range
starts with the most recent commit and walks backwards. A range of 10k
considers only the 10000 most recent commits. A range of zero creates
bitmaps only for branch tips. A range of -1 (the default) does not limit
the range--all commits in the pack are used in the commit selection
process.

Change-Id: Ied92c70cfa0778facc670e0f14a0980bed5e3bfb
Signed-off-by: Terry Parker <tparker@google.com>
2015-10-08 15:17:27 -07:00
Stefan Beller cd1b00e3fc Align docstring for RepoCommand.setRecordRemoteBranch with argument
Change-Id: Ia3aa1130795d162e482b4088f190956d70857244
Signed-off-by: Stefan Beller <sbeller@google.com>
2015-10-06 14:02:19 -07:00
Jonathan Nieder c2efb5c964 Merge "RepoCommand: Add setRecordRemoteBranch option to record upstream branch" 2015-10-05 19:24:48 -04:00
Stefan Beller cdd7c23446 RepoCommand: Add setRecordRemoteBranch option to record upstream branch
On a server also running Gerrit that is using RepoCommand to
convert from an XML manifest to a git submodule superproject
periodically, it would be handy to be able to use Gerrit's
submodule subscription feature[1] to update the superproject
automatically between RepoCommand runs as changes are merged
in each subprojects.

This requires setting the 'branch' field for each submodule
so that Gerrit knows what branch to watch.  Add an option to
do that.

Setting the branch field also is useful for plain Git users,
since it allows them to use "git submodule update --remote" to
manually update all submodules between RepoCommand runs.

[1] https://gerrit-review.googlesource.com/Documentation/user-submodules.html

Change-Id: I1a10861bcd0df3b3673fc2d481c8129b2bdac5f9
Signed-off-by: Stefan Beller <sbeller@google.com>
2015-10-05 16:01:11 -07:00
Matthias Sohn 0baf81a191 Merge "[ignore rules] fix for handling unmatched '[' brackets" 2015-10-02 02:12:19 -04:00
Matthias Sohn d652a6bfd7 Merge branch 'stable-4.1'
* stable-4.1:
  pgm: Open RevWalk and TreeWalk in try-with-resource
  ant: Open Repository and Git in try-with-resource
  pgm: Create instances of Git in try-with-resource
  FanoutBucket: Create ObjectInserter.Formatter in try-with-resource
  Fix compiler warnings in DiffFormatter.writeGitLinkText

Change-Id: I448ecc9a1334977d9f304dd61ea20c7a8e692b10
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-10-02 01:34:59 +02:00
David Pursehouse 581698331a FanoutBucket: Create ObjectInserter.Formatter in try-with-resource
To prevent potential resource leak.

Change-Id: Ife09be2822bc476199f10da8d1eb7ccc8da05b79
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
2015-10-01 15:17:18 +09:00
Matthias Sohn 100179a101 Use japicmp instead of clirr to detect API changes
Clirr doesn't support Java 8 hence use japicmp instead.
See https://github.com/siom79/japicmp

Change-Id: If4b30a6d6aa849b4d6b3b0c900558c609822840c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-10-01 01:41:52 +02:00
Andrey Loskutov 1a2b4e246d [ignore rules] fix for handling unmatched '[' brackets
This patch makes JGit parsing of ignore rules containing unmatched '['
bracket compatible to the Git CLI.

Since '[' starts character group, Git tries to parse the ignore rule as
a shell glob pattern and if the character group is not closed, the glob
pattern is invalid and so the ignore rule never matches anything.

See also http://article.gmane.org/gmane.comp.version-control.git/278699.

Bug: 478490
Change-Id: I734a4d14fcdd721070e3f75d57e33c2c0700d503
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-09-30 18:20:05 +02:00
David Pursehouse a489b6e6b8 Fix compiler warnings in DiffFormatter.writeGitLinkText
- Remove declaration of IOException that is no longer thrown

- Add missing //$NON-NLS-1$ to prevent "Non-externalized string literal"
  warning.

These warnings seem to have been introduced by If13f7b406.

Change-Id: I30058eed31b92067a6ab22e787732b08e29f8d63
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
2015-09-30 18:20:34 +09:00
Matthias Sohn 6664bc6f1c Prepare 4.2.0-SNAPSHOT builds
Change-Id: If559d3565b1f84c93a533e1ce18d5293605d1950
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-09-28 14:48:41 +02:00
Matthias Sohn 729f085ac2 Prepare 4.1.1-SNAPSHOT builds
Change-Id: I035f3a8d0f0de86e8b8f00e668be5ce008402e82
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-09-28 13:37:39 +02:00
Matthias Sohn fdfd6e2872 JGit v4.1.0.201509280440-r
Change-Id: I9a536870b9f5c1247c52d6c976a954115982fa1c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-09-28 10:36:08 +02:00
Matthias Sohn 9e5380e7fb Use java.nio.file consistently in FS
Since 4.0 we require Java 7 so there is no longer a need to override the
following methods in FS_POSIX, FS_Win32, FS_Win32_Cygwin
- lastModified()
- setLastModified()
- length()
- isSymlink()
- exists()
- isDirectory()
- isFile()
- isHidden()
Hence implement these methods in FS and remove overrides in subclasses.

Change-Id: I5dbde6ec806c66c86ac542978918361461021294
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-09-27 22:24:06 +02:00
Andrey Loskutov d35245e906 Deprecate FileUtil and move the code to FileUtils
As discussed on https://git.eclipse.org/r/53836 it does not make sense
to have two similar utility classes in same package with intersecting
functionality. To not break the API, all methods from FileUtil are
copied to FileUtils, all FileUtil API is made deprecated and redirecting
now to FileUtils. Moved simple methods which are available in Java 7 API
are made package private and can be removed at any point later entirely
(right now they are in use).

Bug: 475070
Change-Id: Idffcf9840496c448173af7c052d8898ada68e27b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-09-27 03:14:06 +02:00
Andrey Loskutov 41a972cd1e [performance] Cache platform name in SystemReader
SystemReader.isMacOs() and SystemReader.isWindows() return values are
unlikely to change during the JVM lifetime (except tests). Don't read
system properties each time the methods are called, just use previously
calculated value.

Change-Id: I495521f67a8b544e7b7247d99bbd05a42ea16d20
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-09-27 02:04:57 +02:00
Andrey Loskutov 1abd51d953 [ignore rules] fix for backslash handling
An attempt to re-implement not well documented Git CLI behavior for
patterns with backslashes.

It looks like Git silently ignores all \ characters in ignore rules, if
they are NOT covered by 3 cases described in [1]:

{quote}
1) ... Put a backslash ("\") in front of the first hash for patterns
that begin with a hash.
...
2) Trailing spaces are ignored unless they are quoted with backslash
("\").
...
3) Put a backslash ("\") in front of the first "!" for patterns that
begin with a literal "!", for example, "\!important!.txt".
{quote}

Undocumented also is the fact that backslash itself can be escaped by
backslash.

So \h\e\l\l\o\.t\x\t rule matches hello.txt and a\\\\b a\b in Git CLI.

Additionally, the glob parser [2] knows special meaning of backslash:

{quote}
One can remove the special meaning of '?', '*' and '[' by preceding
them by a backslash, or, in case this is part of a shell command
line, enclosing them in quotes.  Between brackets these characters
stand for themselves.  Thus, "[[?*\]" matches the four characters
'[', '?', '*' and '\'.
{quote}

[1] https://www.kernel.org/pub/software/scm/git/docs/gitignore.html
[2] http://man7.org/linux/man-pages/man7/glob.7.html

Bug: 478065
Change-Id: I3dc973475d1943c5622103701fa8cb3ea0684e3e
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-09-26 18:16:42 +02:00
Andrey Loskutov 4b7daecf4a [ignore rules] Fix for character group matcher
Currently we fail to properly recognize character group if the pattern
before character group contains opening bracket.

See comment from Sebastien Arod on https://git.eclipse.org/r/56678/

Change-Id: I70d3657a2a328818ea2bdc1409d18ecb3a85825b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-09-26 11:34:02 +02:00
Matthias Sohn 6715c36100 Merge "Show submodule difference as a hunk" 2015-09-25 17:48:04 -04:00
Jacob Keller a28ba3f2c7 Show submodule difference as a hunk
Current DiffFormat behavior regarding submodules (aka git links) is
incorrect. The "Subproject commit <sha1>" appears as part of the diff
header, rather than as its own hunk.

--> From JGit implementation
 diff --git a/plugins/cookbook-plugin b/plugins/cookbook-plugin
 index b9d3ca8..ec6ed89 160000
 --- a/plugins/cookbook-plugin
 +++ b/plugins/cookbook-plugin
 -Subproject commit b9d3ca8a65030071e28be19296ba867ab424fbbf
 +Subproject commit ec6ed89c47ba7223f82d9cb512926a6c5081343e

--> From C Git 2.5.2
 diff --git a/plugins/cookbook-plugin b/plugins/cookbook-plugin
 index b9d3ca8..ec6ed89 160000
 --- a/plugins/cookbook-plugin
 +++ b/plugins/cookbook-plugin
 @@ -1 +1 @@
 -Subproject commit b9d3ca8a65030071e28be19296ba867ab424fbbf
 +Subproject commit ec6ed89c47ba7223f82d9cb512926a6c5081343e

The current way of processing submodules results in no hunk header and
includes the contents of the hunk as part of the headers. To fix this, we
can't just have our writeGitLinkDiffText output the hunk header. We have
to change the flow so that the raw text gets parsed as a diff. The easiest
way to do this is to fake the RawText in the FormatResult when we have a
GITLINK.

It should be noted that it seems possible for there to be a difference
between a GITLINK and a non-GITLINK, but I don't think this can happen in
practice, so I don't think we need to worry too much about it.

This patch also fixes up the test for GitLink headers, as the test was
for the old behavior. My setup has 3 other failing tests which may or
may not be the result of environmental changes. However, the same tests
fail without this commit, so I do not believe they are related.

Bug: 477759
Change-Id: If13f7b406904fad814416c93ed09ea47ef183337
Signed-off-by: Jacob Keller <jacob.keller@gmail.com>
2015-09-24 17:52:36 -04:00
Andrey Loskutov 97ef1fe893 Properly support special regex characters in ignore rules
Ignore rules should escape $^(){}+| chars if using regular expressions,
because they should be treated literally if they aren't part of a
character group.

Bug: 478055
Change-Id: Ic7276442d7f8f02594b85eae1ef697362e62d3bd
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-09-24 16:04:57 +02:00
Christian Halstrick f4596284a0 Fix endless loop in ObjectChecker for MacOS
Bug: 477090
Change-Id: I0ba416f1cc172a835dd2723ff7fa904597ffd097
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-09-17 23:22:14 +02:00
Hugo Arès 868899982b Fix warnings about missing serialVersionUID
Change-Id: Ieff64896aebeab793ff08ab89f10d5ccaee66021
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
2015-09-17 10:23:34 +02:00
Shawn Pearce b46c446395 UploadPack: Verify clients send only commits for shallow lines
If a client mistakenly tries to send a tag object as a shallow line
JGit blindly assumes this is a commit and tries to parse the tag
buffer using the commit parser. This can cause an obtuse error like:

  InvalidObjectIdException: Invalid id: t c0ff331234...

The "t" comes from the "object c0ff331234..." line of the tag tring
to be parsed as though it where the "tree" line of a commit.

Run any client supplied shallow lines through the RevWalk to lookup
the object types. Fail fast with a protocol exception if any of them
are non-commit.

Skip objects not known to this repository. This matches behavior
with git-core's upload-pack, which sliently skips over any shallow
line object named by the client but not known by the server.

Change-Id: Ic6c57a90a42813164ce65c2244705fc42e84d700
2015-09-14 12:53:34 -07:00
Shawn Pearce db0adc1e8c S3 transport: Fix check for tmpdir
Properties.containsKey() is the correct call here; contains() was testing
if a value is present but the key is what was meant.

Change-Id: Ice72c9f4388583e18cf8aca6e837cc4299fd07fd
2015-09-10 14:18:50 -07:00
Patrick Steinhardt 9b1deadcb4 URIish: fall back to host as humanish name
When we have a URI that contains an empty path component (that is
it only contains a "/") we want to fall back to the host as
humanish name.

This change is according to the behavior of upstream git, which
falls back on the hostname when guessing directory names for
newly cloned repositories (see [1] for the discussion).

[1] http://article.gmane.org/gmane.comp.version-control.git/274669

Change-Id: I44400c6ab72a2722d2155d53d63671bd867d6c44
Signed-off-by: Patrick Steinhardt <ps@pks.im>
2015-09-10 13:21:41 +02:00
Dave Borowitz 45e9e28ad9 PushCertificateStore: Don't add no-op command to batch
If no refs match the input list and we are writing to a batch,
the returned new commit from write() will match the current commit.
Adding a command to the batch for this case is harmless as it will
succeed, but it's more straightforward to just skip adding a command
in this case.

Add tests or the combination of saving matching refs and saving to a
batch.

Change-Id: I6837389b08e6c80bc2d4c9e9c506d07293ea5fb2
2015-09-02 15:17:10 -04:00
Matthias Sohn 4a11534c25 Restore lazy Bundle-ActivationPolicy removed in 3a4a5a4e
This header was removed unintentionally from some bundles in
3a4a5a4e57. Restore it to ensure lazy
activation of bundles.

Change-Id: I1f841f978fb93278e3ec0533a01f1363510dd976
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-08-31 22:18:54 +02:00
Matthias Sohn 3a4a5a4e57 Update uses-clauses in OSGi manifests
In Bug 476164 it was reported that EGit doesn't start when the platform
comes with jsch 0.1.51 while this version of EGit/JGit brings jsch
0.1.53. This could be caused by outdated uses-clauses. Hence recompute
them using PDE tooling.

Bug: 476164
Change-Id: I185ba097884ead9cd034eba842bd3bf34181a99b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-08-31 11:34:06 +02:00
Matthias Sohn 05acf1c62f Use java.io.File to check existence of loose objects in ObjectDirectory
It was reported in [1] that 197e3393a5 led
to a performance regression in a BFG benchmark. Analysis showed that
this is caused by the exists() method in FS_POSIX, now overriding the
default implementation in FS. The default implementation of FS.exists()
uses java.io.File.exists(), while the new implementation in FS_POSIX
uses java.nio.file.Files.exists() - by simply removing the override in
FS_POSIX, performance was restored.

Profiling showed that java.nio.file.Files.exists() is substantially
slower than java.io.File.exists(), to the point where the exists() call
doubles the average cost of a call to
ObjectDirectory.insertUnpackedObject() - which the BFG uses a lot,
because it's rewriting history. Average times measured on Ubuntu were:

java.io.File.exists() - 4 microseconds
java.nio.file.Files.exists() - 60 microseconds

The loose object exists test should be using java.io.File and not FS.
ObjectDirectory uses FS.resolve() to traverse symlinks to objects but
then once inside objects all 256 sharded directories should be real
directories, and the object files should be real files, not dangling
symlinks. java.io.File.exists() is sufficient here, and faster.

Change ObjectDirectory to use File.exists() once its computed the File
handle.

This does mean JGit cannot run ObjectDirectory code on an abstract
virtual filesystem plugged into NIO2. If you really want to run JGit on
an esoteric non-standard filesystem like "in memory" you should look at
the DFS storage backend, which has fewer abstraction points to deal
with. Or write your own from scratch.

[1] https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg02954.html

Change-Id: I74684dc3957ae1ca52a7097f83a6c420aa24310f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-08-28 13:09:50 +02:00
Matthias Sohn 3abf35bc0f Fix warnings on FileUtils.isStaleFileHandle()
- add @since annotation for new API method
- silence non-externalized String warning

Change-Id: I864176ced64e9569e7f2cdf8f777720655bfc578
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-08-26 23:30:12 +02:00
Andrey Loskutov 35d45abfb2 Deprecate redundant FileUtil.delete(File), use FileUtils instead
Bug: 475070
Change-Id: I6dc651f4b47e1b2c8d7954ec982e21ae6bb5f7a6
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-08-26 23:25:38 +02:00
Martin Fick 06b446057c Handle stale file handles on packed-refs file
On a local filesystem the packed-refs file will be orphaned if it is
replaced by another client while the current client is reading the old
one. However, since NFS servers do not keep track of open files, instead
of orphaning the old packed-refs file, such a replacement will cause the
old file to be garbage collected instead.  A stale file handle exception
will be raised on NFS servers if the file is garbage collected (deleted)
on the server while it is being read.  Since we no longer have access to
the old file in these cases, the previous code would just fail. However,
in these cases, reopening the file and rereading it will succeed (since
it will reopen the new replacement file).  So retrying the read is a
viable strategy to deal with stale file handles on the packed-refs file,
implement such a strategy.

Since it is possible that the packed-refs file could be replaced again
while rereading it (multiple consecutive updates can easily occur with
ref deletions), loop on stale file handle exceptions, up to 5 extra
times, trying to read the packed-refs file again, until we either read
the new file, or find that the file no longer exists. The limit of 5 is
arbitrary, and provides a safe upper bounds to prevent infinite loops
consuming resources in a potential unforeseen persistent error
condition.

Change-Id: I085c472bafa6e2f32f610a33ddc8368bb4ab1814
Signed-off-by: Martin Fick<mfick@codeaurora.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-08-26 22:53:07 +02:00
Matthias Sohn cc50ec2d87 Merge "Add public isStaleFileHandle() API, improve detection." 2015-08-26 16:16:30 -04:00
Martin Fick cb08dd8b14 Add public isStaleFileHandle() API, improve detection.
Add a public API to the FileUtils to determine if an IOException is a
stale NFS file handle exception.  This will make it easier to detect
such errors, and interpret them consistently throughout the codebase.
This new API is a bit more lenient in its detection than the previous
detection, and should be able to detect some errors which previously
were not identified as stale file handle exceptions because they had the
word NFS in the error message.  Adjust the packfile handling code to use
this new API for detection.

Change-Id: I21f80014546ba1afec7335890e5ae79e7f521412
Signed-off-by: Martin Fick<mfick@codeaurora.org>
2015-08-25 08:19:54 -06:00
Andrey Loskutov b5941c74e5 Set "potentialNullReference" to "error" level and fixed all issues
There should be no functional change, the logic updated only to make
code simple so that compiler can understand what is going for. Removed
all @SuppressWarnings("null") annotations since they cannot be used if
"org.eclipse.jdt.core.compiler.problem.potentialNullReference" option is
set to the "error" level.

Bug: 470647
Change-Id: Ie93c249fa46e792198d362e531d5cbabaf41fdc4
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-22 13:53:41 +02:00
Andrey Loskutov 847b3d1258 Enable annotation based NPE analysis in jgit
Bug: 470647
Change-Id: I14d1983bb7c208faeffee0504e0567e38d8a89f3
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-20 21:10:04 +02:00
Andrey Loskutov 8b38c7db42 Restored obsoleted createSymLink/readSymLink in FileUtil
Bug: 475070
Change-Id: I425ad842dc26b55f747f192348398a3912c0ca6b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-18 20:01:15 +02:00
Matthias Sohn 548ba66a37 Use NIO2 to implement FileUtils.rename() and expose options
- use NIO2's Files.move() to reimplement rename()
- provide a second method accepting CopyOptions which can be used to 
  request atomic move.

Change-Id: Ibcf722978e65745218a1ccda45344ca295911659
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-08-17 23:25:20 +02:00
Andrey Loskutov 2e5c7c5db4 Move createSymLink/readSymLink to FileUtils
Bug: 475070
Change-Id: I258f4bf291e02ef8e6f867b5d71c04ec902b6bcb
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-17 16:06:15 +02:00
Christian Halstrick 91b1ab90e2 Merge "Change FS not to throw NPE when facing InMemory databases" 2015-08-17 03:54:25 -04:00
Shawn Pearce 1d7d0f95e0 Expose the set of root commits in PackStatistics
Root commits are commits with zero parents.  If a commmit has no
parents it is the first commit in the repository.  In general the root
commits should be unique for any given project, as the first commit
will be created at a different time, by a different user with its own
message.  These root commits can be used as a "fingerprint" to
identify disjoint histories.

Change-Id: Id891dbc1f17c816cea404569578bb7635ff85cdb
2015-08-16 14:10:16 -07:00
Christian Halstrick d0d637342b Change FS not to throw NPE when facing InMemory databases
The FS class and the subclasses FS_POSIX assumed in the findHook()
method that every repository has a valid gitDir. But in tests when using
in-memory-repositories this is not true and this method was generating
NPEs. Change the method to return null if no repository directory can be
determined.

Change-Id: I38a4d36dc6452b5dacae3d0dbf562b569ca3c19b
2015-08-16 22:20:06 +02:00
Shawn Pearce f9bd6c1239 Fix NPE in DfsGarbageCollector and further reduce memory
DfsGarbageCollector asks PackWriter for the set of objects packed
after the bitmap index is written out.  This is now null as it was
cleared to release memory. Instead use PackBitmapIndexBuilder to
build the set as it also has the objects.

Reduce memory in PackBitmapIndexBuilder by fully discarding the
ObjectToPack instances. This was the original intent of commit
4bb523475d ("PackWriter: shed memory while creating bitmaps")
but failed as the instances were still held live here.

Switch to BlockList instead of ObjectToPack[]. This allows the
JVM to allocate many smaller arrays instead of one contiguous
array with 5.2M reference pointers. In a tight heap the smaller
allocations are more feasible.

Reduce the initial EWAHCompressedBitmaps for the 4 type maps.  On
average a typical repository is 30% commits, 30% trees and 30% blobs.
These bitmaps are typically very dense.  PackWriter orders objects by
commit, tree, blob when writing the file so these should always be a
very dense run of 1s with some 0s before and after. So even the 1/3rd
allocation is likely too large, but the later trim() will reduce the
internal buffer anyway.

Change-Id: If0b80a31cb00894f1485ff8f53ef7ae5a759a046
2015-08-14 11:13:34 -07:00
Andrey Loskutov bfc3e1262c Cleanup Attributes and remove obsoleted Java7BasicAttributes class
After jgit moved to Java 7 there is no need in an extra
Java7BasicAttributes class. Also all fields of Attributes can be made
final now.

Change-Id: I0be6daf7758189b0eecc4e26294bd278ed8bf7a0
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-13 15:12:05 -04:00
Shawn Pearce f5a5fa28d2 Merge "Remove "experimental" from the description of the pack bitmap index" 2015-08-13 15:07:39 -04:00
Shawn Pearce 4bb523475d PackWriter: shed memory while creating bitmaps
Once bitmap creation begins the internal maps required for packing are
no longer necessary.  On a repository with 5.2M objects this can save
more than 438 MiB of memory by allowing the ObjectToPack instances to
get garbage collected away.

Downside is the PackWriter cannot be used for any further opertions
except to write the bitmap index.  This is an acceptable trade-off as
in practice nobody uses the PackWriter after the bitmaps are built.

Change-Id: Ibfaf84b22fa0590896a398ff659a91fcf03d7128
2015-08-13 10:13:44 -07:00
Shawn Pearce 20d9ab00ae Merge "Do not retain commit body during bitmap generation" 2015-08-13 12:09:04 -04:00
Shawn Pearce 349b0af754 Merge "Bitmap builder: actually compress EWAH bitmaps in memory" 2015-08-13 12:08:31 -04:00
Shawn Pearce 88bffda524 Bitmap builder: actually compress EWAH bitmaps in memory
For construction performance each new EWAHBitmap is allocated at the
roughly worst-case size the bitmap would need if all of the words must
be literal and no run length compression is available.  In practice
this is far larger than is required, wasting heap memory while the
bitmaps are computed.

Trim down each bitmap to its minimum required size. This copies the
internal array to a new smaller array, allowing the GC to reclaim the
prior larger array for reuse.

A single bitmap of 5.2M bits is only 79 KiB of memory without this
trim call but 15,000 such bitmaps is 1.1 GiB. Trimming can help fit
a larger number of bitmaps during processing.

Change-Id: I2bd19a786189db5b01c4c96f209b83de50e10c3b
2015-08-12 23:18:24 -07:00
Shawn Pearce d988540c8e Do not retain commit body during bitmap generation
The bitmap preparer only needs commit graph topology; it does not use
the message body.  Allow the RevWalk to free the body after the commit
has been parsed to save memory.

Change-Id: I97d4a440c9fc313873fd224bd05b9d9e3dc575db
2015-08-12 22:10:35 -07:00
Andrey Loskutov 63bc1d862d Consider original file mode while checking parent ignore rules
The WorkingTreeIterator.isEntryIgnored() should use originally requested
file mode while descending to the file tree root and checking ignore
rules. Original code asking isEntryIgnored() on a file was using
directory mode instead if the .gitignore was not located in the same
directory.

Bug: 473506
Change-Id: I9f16ba714c3ea9e6585e9c11623270dbdf4fb1df
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-12 13:44:58 -04:00
Andrey Loskutov 7979029b43 Clone should create Git instances which are able to close repository
Bug: 474093
Change-Id: I13be133dac7834d1d2b51eb9948a716b8719d057
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-04 13:37:58 +02:00
Christian Halstrick 5c2f52f72b Merge "Clone should close repository after errors in fetch or checkout" 2015-08-04 07:34:22 -04:00
Andrey Loskutov a28ae3995f containsGitModulesFile() should not crash on bare repository
Change-Id: Iba7e4674b3d33c730613a6ac703977f48b015853
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-03 07:58:52 -04:00
Andrey Loskutov 1309526564 Clone should close repository after errors in fetch or checkout
Bug: 474093
Change-Id: Ia0a1478260b94a71a947aa8c04ee0c836d390aec
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-08-02 23:46:09 +02:00
Andrey Loskutov 204da3969e Fixed javadoc: the getBranch() and getFullBranch() can return null
Change-Id: I284a1b5f8220f68496f992b0e97e6934e03ae616
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-07-29 18:04:50 -04:00
Dave Borowitz 94812ef1e9 PushCertificate: Omit null pushee from text representation
Change-Id: Ie9546f2e0e9ee62e0a3c919572153b6076355195
2015-07-28 10:53:07 -07:00
Christian Halstrick 1196dd0643 Fix ResolveMerger when files should be replaced by folders
When during Merge for a certain path OURS & BASE contains a file and
THEIRS contains a folder there was a bug in JGit leading to unnecessary
conflicts. This commit fixes it and adds a test for this situation.
 
Bug: 472693
Change-Id: I71fac5a6a2ef926c01adc266c6f9b3275e870129
Also-by: Clemens Buchacher <drizzd@aon.at>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-07-23 11:21:53 +02:00
Matthias Sohn c92f0ba096 Merge "Fix typos in javadoc of ObjectId.equals()" 2015-07-21 18:37:53 -04:00
Andrey Loskutov 4e7639bb65 Don't keep empty ignore rules in the ignore node list
Change-Id: Icd893dfaba06561bbe5cc60ebf866ec5d8301c22
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-07-21 01:23:40 +02:00
Andrey Loskutov dfed946f10 Allow leading/trailing spaces in ignore rules
According to [1] leading spaces are allowed in ignore rules and trailing
spaces are allowed too if they are escaped via backslash.

[1] https://www.kernel.org/pub/software/scm/git/docs/gitignore.html

Bug: 472762
Change-Id: I5e3ae5599cb9e5d80072f38c82c20cbc9475a18a
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-07-21 01:23:40 +02:00
Andrey Loskutov b98c84eb88 Don't crash while parsing ignore patterns
Catch unexpected PatternSyntaxException and convert it to
InvalidPatternException. Log such errors, do not silently ignore them.

Bug: 463581
Change-Id: Id0936d9816769ec0cfae1898beda0f7a3c146e67
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-07-21 01:23:37 +02:00
Andrey Loskutov 08641ea413 Allow leading literal '#' and '!' in ignore rules if they are escaped
According to [1] backslash can escape leading special characters '#' and
'!' in ignore rules, so that they are treated literally.

[1] https://www.kernel.org/pub/software/scm/git/docs/gitignore.html

Bug: 463581
Change-Id: I4c02927413a9c63ea5dbf2954877080d902ec1b2
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-07-21 00:47:28 +02:00
Andrey Loskutov 1d295cb7fb Don't trim trailing space if it is escaped with backslash
According to [1] backslash can escape trailing space in ignore rules.

[1] https://www.kernel.org/pub/software/scm/git/docs/gitignore.html

Bug: 463581
Change-Id: I9cf13f8775cb49f0b6d61cfd3ca3fd6d665fccd8
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-07-21 00:47:28 +02:00
Andrey Loskutov f19b1f2d07 Consider only escaping backslash for regular expressions in ignore rules
While checking if we should consider an ignore rule without '[]'
brackets as a regular expression, check if the backslash escapes one of
the glob special characters '?', '*', '[', '\\'. If not, backslash is
not a part of a regex and should be treated literally.

Bug: 463581
Change-Id: I85208c7f85246fbf6c5029ce3c8b7bb8f4dbd947
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-07-21 00:47:28 +02:00
Terry Parker 4d6046da77 Remove "experimental" from the description of the pack bitmap index
Change-Id: I940701093eb6c6d26e147b79de77997b0df481a0
Signed-off-by: Terry Parker <tparker@google.com>
2015-07-20 11:28:58 -07:00
Matthias Sohn f53306bf2a Fix typos in javadoc of ObjectId.equals()
Change-Id: Ic2503a2c3c616934d923937fdf8ceb4ae57b7028
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-07-20 14:09:34 +02:00
Jonathan Nieder f8564c7601 DFS: Remove now-redundant getOneRef
Instead of reimplementing exactRef in a separate, private method,
use it directly.

Change-Id: I0fc06fc46eef0e36e571a6ef622f38dc2aa59038
2015-07-16 14:50:42 -07:00
Shawn Pearce d0e47a99aa DFS: Override exactRef() with optimal version
This avoids scanning a directory to find all names and
then weeding down to the requested name.

Change-Id: I36ce31febad6e881182bc3a278c7ed35cc04cbda
2015-07-16 14:31:50 -07:00
Dave Borowitz 39dc898dca PushCertificate: Add toTextWithSignature() method
Change-Id: Ic0129373cc0c0321ffa43dc11a658d4f891ec2c2
2015-07-15 18:36:45 -07:00
Dave Borowitz 74a57f3744 PushCertificateParser: Add fromString method
Change-Id: I74c3f65a9ff297f708d996a4c138456a31a466b8
2015-07-15 18:09:42 -07:00
Dave Borowitz c7c5fe718d BaseReceivePack: Add method to override the push cert
Change-Id: Id912f45c933c78a85b245fa7b6edebbeb40b009c
2015-07-15 18:09:42 -07:00
Dave Borowitz d8b9c5145b BaseReceivePack: Don't throw from getPushCertificate()
Rather than lazily parsing the push in this method, parse it at the
end of recvCommands(), which already contains the necessary try/catch
for handling this error. This allows later callers to avoid having to
handle this condition superfluously.

Change-Id: I5dcaf1a44bf4e321adc281e3381e7e17ac89db06
2015-07-15 18:09:42 -07:00
Dave Borowitz d7377877e0 PushCertificateStore: Return boolean from batch save methods
Change-Id: I9730cb4f60c60ee6d5a7a156a0b6a53f79309ec3
2015-07-15 18:09:42 -07:00
Dave Borowitz 5706c8e38e Allow saving push certs on a subset of refs
Consider a BatchRefUpdate produced by Gerrit Code Review, where the
original command pushed over the wire might refer to
"refs/for/master", but that command is ignored and replaced with some
additional commands like creating "refs/changes/34/1234/1". We do not
want to store the cert in "refs/for/master@{cert}", since that may
lead someone looking to the ref to the incorrect conclusion that that
ref exists.

Add a separate put method that takes a collection of commands, and
only stores certs on those refs that have a matching command in the
cert.

Change-Id: I4661bfe2ead28a2883b33a4e3dfe579b3157d68a
2015-07-15 18:09:41 -07:00
Dave Borowitz f946ed1d97 PushCertificateStore: Add method to save in batch
Change-Id: I8bfaee1a52d368ffe2cd7e8af1754a5261569078
2015-07-13 08:30:11 -07:00
Andrey Loskutov b4ecb88367 Fixed @since tags on new API
Change-Id: I77bc2ec486ffe4c8b864663d103d8cf39c3f35e1
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
2015-07-12 18:44:08 +02:00
Dave Borowitz d5a71e9ca3 Store push certificates in refs/meta/push-certs
Inspired by a proposal from gitolite[1], where we store a file in
a tree for each ref name, and the contents of the file is the latest
push cert to affect that ref.

The main modification from that proposal (other than lacking the
out-of-git batching) is to append "@{cert}" to filenames, which allows
storing certificates for both refs/foo and refs/foo/bar. Those
refnames cannot coexist at the same time in a repository, but we do
not want to discard the push certificate responsible for deleting the
ref, which we would have to do if refs/foo in the push cert tree
changed from a tree to a blob.

The "@{cert}" syntax is at least somewhat consistent with
gitrevisions(7) wherein @{...} describe operators on ref names.

As we cannot (currently) atomically update the push cert ref with the
refs that were updated, this operation is inherently racy. Kick the can
down the road by pushing this burden on callers.

[1] cf062b8bb6/contrib/hooks/repo-specific/save-push-signatures

Change-Id: Id3eb32416f969fba4b5e4d9c4b47053c564b0ccd
2015-07-10 13:16:37 -07:00
Yuxuan 'fishy' Wang 217b2a7cc5 Add setTargetBranch in RepoCommand.
This will allow us to write the super project in a branch other than
master.

Change-Id: I578ed9ecbc6423416239e31ad644531dae9fb5c3
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>
2015-07-10 11:39:26 -07:00
Dave Borowitz 19b45dab71 PushCertificateParser: Move reader parsing to instance method
Change-Id: I8ab5fb97303e25b28187375b6f9ff28f918bdd43
2015-07-09 12:02:45 -07:00
Dave Borowitz 1f2827ecaa PushCertificateParser: Make constructor public
Primarily to aid in writing end-to-end tests, which must be written
outside this package as we don't currently have a Bouncy Castle
dependency.

Change-Id: Ia24179773140d9ecc616b3a6b2e630c85d800157
2015-07-09 11:50:11 -07:00
Dave Borowitz 6e4e34bb9e Report PackProtocolExceptions to client during receive-pack
We have done this since forever with the "wanted old new ref" error,
so let's do it for other such errors thrown in the same block as well.

Change-Id: Ib3b1c7f05e31a5b3e40e85eb07b16736920a033b
2015-07-09 11:41:24 -07:00
Dave Borowitz a643aaaa4a PushCertificate: implement hashCode/equals/toString
Change-Id: Ib588a3f47492cee5e5e6274a3b088678919a0fa0
2015-07-09 11:15:41 -07:00
Dave Borowitz 8293c770b5 PushCertificateParser: Make pushee optional
When pushing to an HTTP server using the C git client, I observed a
certificate lacking a pushee field. Handle this gracefully in the
parser.

Change-Id: I7f3c5fa78f2e35172a93180036e679687415cac4
2015-07-09 11:05:45 -07:00
Dave Borowitz f22f33f5bf PushCertificateParser: Change error string for invalid field
The old one didn't take an argument; the new one includes the field
name.

Change-Id: I175dc75dcdd05ba6fdec315c04c79550cfdc2342
2015-07-09 10:56:02 -07:00
Dave Borowitz 618f213da0 PushCertificateParser: Add method for parsing from a stream
We intend to store received push certificates somewhere, like a
particular ref in the repository in question. For reading data back
out, it will be useful to read push certificates (without pkt-line
framing) in a streaming fashion.

Change-Id: I70de313b1ae463407b69505caee63e8f4e057ed4
2015-07-09 10:55:52 -07:00
Dave Borowitz dac6ae3d17 IO: Add a method for reading lines
Change-Id: Ib7be76aa7ac889354ad4782e2b64d4221a0e25b9
2015-07-08 10:29:11 -07:00
Dave Borowitz 469734bf87 BaseReceivePack: Treat all LFs as optional
Discussion on the git mailing list has concluded[1] that the intended
behavior for all (non-sideband) portions of the receive-pack protocol
is for trailing LFs in pkt-lines to be optional. Go back to using
PacketLineIn#readString() everywhere.

For push certificates specifically, we agreed that the payload signed
by the client is always concatenated with LFs even though the client
MAY omit LFs when framing the certificate for the wire. This is still
reflected in the implementation of PushCertificate#toText().

[1] http://thread.gmane.org/gmane.comp.version-control.git/273175/focus=273412

Change-Id: I817231c4d4defececb8722142fea18ff42e06e44
2015-07-07 15:44:17 -04:00
Dave Borowitz 59b000a672 BaseReceivePack: More validation during parseCommand
Change-Id: I25f3a5582a45dd0ec8f78f5daf74c2203797a184
2015-07-07 15:44:17 -04:00
Matthias Sohn 91e17b0080 Fix non-escaped quotes in JGitText.properties
In most texts we use "cannot" hence instead of escaping the apostroph in
"can't" use "cannot".

Bug: 471796
Change-Id: Icda5b4db38076789d06498428909306aef3cb68b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-07-07 08:24:43 +02:00
Jonathan Nieder 761f070866 Throw InvalidObjectIdException from ObjectId.fromString("tooshort")
ObjectId.fromString already throws InvalidObjectIdException for most
malformed object ids, but for this kind it previously threw
IllegalArgumentException.  Since InvalidObjectIdException is a child of
IllegalArgumentException, callers that catch IllegalArgumentException
will continue to work.

Change-Id: I24e1422d51607c86a1cb816a495703279e461f01
Signed-off-by: Jonathan Nieder <jrn@google.com>
2015-07-06 12:41:06 -07:00
Jonathan Nieder 0c7ad12c76 Avoid double-colon in InvalidObjectIdException description
The invalidId message in JGitText and the asAscii bad id both contain a
colon, so the resulting message would say

	Invalid id: : a78987c98798ufa

Fix it by keeping the colon in the translated message and not adding
another colon programmatically.

Noticed by code inspection.

Change-Id: I13972eebde27a4128828e6c64517666f0ba6288b
Signed-off-by: Jonathan Nieder <jrn@google.com>
2015-07-06 12:34:31 -07:00
Matthias Sohn 7e0d28e924 Merge branch 'stable-4.0'
Change-Id: I5c965206ad10bababe366a51ab7c33a8836a7868
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-06-24 14:46:51 +02:00
Matthias Sohn 176c4b4d5e Prepare 4.0.2-SNAPSHOT builds
Change-Id: I645cacfdde21aa28aa2e17c10dec0576b170ed0e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-06-24 13:51:21 +02:00
Matthias Sohn 90f36e5002 JGit v4.0.1.201506240215-r
Change-Id: Ib7713b657e7812b0debd72bb4eece0daa187e80d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-06-24 08:17:29 +02:00
Tobias Oberlies d34314644e API to remove repositories from RepositoryCache
Add methods that allow to unregister repositories from the
RepositoryCache individually.

Bug: 470234
Change-Id: Ib918a634d829c9898072ae7bdeb22b099a32b1c9
Signed-off-by: Tobias Oberlies <tobias.oberlies@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-06-23 14:30:18 +02:00
Christian Halstrick 6b65adca2d Add a grace period for packfiles during GC
For loose objects an expiration date can be set which will save too
young objects from being deleted. Add the same for packfiles. Packfiles
which are too young are not deleted.

Bug: 468024
Change-Id: I3956411d19b47aaadc215dab360d57fa6c24635e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2015-06-22 11:18:31 +02:00
Dave Borowitz d2fbbc910a SignedPushConfig: Allow setting a custom nonce generator impl
Change-Id: Ic0156a7d65d99881ef27801fcce7754594c436f0
2015-06-18 13:24:04 -04:00
Dave Borowitz ea21f17f29 PackCertificateParser: return null if nothing was received
Add test for this case in both the enabled and disabled cases.

Change-Id: If9d12192a2dc9f9dd1eac9844b5a7b0edadc0b34
2015-06-18 13:13:43 -04:00