* stable-6.5:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I7272a22451c0de6b4770767e7bb4e24c81518c20
* stable-6.4:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I2951d01f5f4581bee20079508cd8ee6ca8554f1f
* stable-6.3:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I4b94a2b79941c085fa2f62246e8e879aaa85cd3f
* stable-6.2:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I22b89bf00dcef26b2096d25397aa9a57a745a92b
* stable-6.1:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: Ib4e4fe407dce334c7537bf278baa39db93aa2f09
* stable-6.0:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I0c9c0b3c206cac03a93b30eda348177a4de35c36
* stable-5.13:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: Ifeaa4b4f0c5944d4ecd3042be429833ff72b43ed
Change Ide3445e652 introduced the `--pack-kept-objects` option to GC for
including the objects contained in the locked packfiles during the
repack phase.
Whilst this allowed to explicitly pass a command line argument to the
jgit gc program, it did not allow the option to be read from
configuration.
Allow the pack kept objects option to be configured exactly as C-Git
documents [1], by introducing a new `repack.packKeptObjects`
configuration.
`repack.packKeptObjects` defaults to `true`, when the
`pack.buildBitmaps` is `true` (which is the default case), `false`
otherwise.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-repackpackKeptObjects
Bug: 582292
Change-Id: Ia931667277410d71bc079d27c097a57094299840
Packfiles having an equivalent .keep file are associated with in-flight
pushes that haven't been completed, with potentially a set of git
objects not yet referenced by a ref.
If the Git client is not up-to-date, it may result in pushing a
packfile, generating a <packfile>.keep on the server, which
may also contain existing commits due to the lack of Git protocol
negotiation in the git-receive-pack.
The Git protocol negotiation is the phase where the client and the
server exchange the list of refs they have for trying to find a common
base and minimise the amount of objects to be transferred.
The repack phase in GC was previously skipping all objects that were
contained in all packfiles having a <packfile>.keep file associated
(aka "locked packfiles"), which did not take into consideration the
fact that excluding the existing commits would have resulted in the
generation of an invalid bitmap file.
The code for excluding the objects in the locked packfiles was written
well before the bitmap was introduced, hence could not consider a use
case that did not exist at that time.
However, when the bitmap was introduced, the exclusion of locked
packfiles was not changed, hence creating a potential problem.
The issue went unnoticed for many years because the bitmap generation
was disabled when JGit noticed any locked packfiles; however, the
bitmaps are enabled again since Id722e68d9f , and the the issue is now
visible and is impacting the GC repack phase.
Introduce the '--pack-kept-objects' option in GC for including the
objects contained in the locked packfiles during the repack phase,
which is not an issue because of the following:
- If there are any existing commits duplicated in the packfiles
they will be just considered once anyway because the repack doesn't
generate duplicates in the output packfile.
- If there are any new commits that do not have any ref pointing to
them, they will be automatically excluded from the output repacked
packfile.
The same identical solution is adopted in the C implementation of git
in repack.c.
Because the locked packfile is not pruned, any new commits not pointed
by any refs will remain in the repository and there will not be any
accidental pruning or object loss as it is today before this change.
As a side-effect of this change, it is now potentially possible to still
have duplicate BLOBs after GC when the keep packfile contained existing
objects. However, it is way better to keep the duplication until the
next GC phase rather than omitting existing objects from repacking and,
therefore generating an invalid bitmap and incorrect packfile.
Bug: 582292
Bug: 582455
Change-Id: Ide3445e652fcf256a7912f881cb898897c99b8f8
When checking out a file into the working tree ensure that all parent
directories of the file below the working tree root are actually
directories and do exist before we try to create the file.
When multiple files are to be checked out (or even a whole tree), this
may check the same directories over and over again. Asking the file
system every time for file attributes is a potentially expensive
operation. As a remedy, introduce an in-memory cache of directory
states for a particular check-out operation.
Apply the same fix also in the ResolveMerger, which may also check out
files, and also in the PatchApplier. In PatchApplier, also validate
paths.
Change-Id: Ie12864c54c9f901a2ccee7caddec73027f353111
Signed-off-by: Thomas Wolf <twolf@apache.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-6.5:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: Id2e49252a9dc268210c9439848e77604885371aa
* stable-6.4:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: Idb6dd6160e023673e3650653a15f6b1c540de96e
* stable-6.3:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I0bccc36d9cc9a36f1be9b1562df35ce3a0e95eee
* stable-6.2:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I589ed444b5cbfc5b073cac91323e2cc97ab98087
* stable-6.1:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I5b16c3b613a95b7f28c8f6ac0b20c4c593759cea
* stable-6.0:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: Ib08037f6055dac1776e38cfb4ff8c88a50ad3e60
* stable-5.13:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I1f50995d9d9c592ec0e02a04e0e409440b49f9f3
The packfiles with the .keep extensions are meant to prevent
a packfile from being processed or removed during GC.
From the point of view of the GC process then, the associated
packfile should be completely transparent:
- it should not included in the repacked file
- it should not pruned
- its objects should be left untouched, even if unreachable
- the GC process, including the bitmap generation should continue
as usual, as the the packfiles with .keep file did not exist
Add one explicit test for making sure that the management
of .keep file is also transparent to the generation of bitmaps,
which are still generated if a .keep file exists.
Bug: 582039
Change-Id: I14f6adc3f961c606fbc617e51ea6ed6e2ef8604f
In findGraphPosition, when there is no object whose OID starts with
the first byte of the sought OID, low equals high. This violates an
invariant of the loop, and when the sought OID is lexicographically
greater than every other OID in the repository, causes an
ArrayIndexOutOfBoundsException (because we're trying to read outside the
list of OIDs).
Therefore, check the "low < high" condition at the start of the loop,
not only after the first iteration.
Change-Id: Ic8ac198c151bd161c4996b9e7cb6e6660f151733
Helped-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
PackWriter offers the ability to write out the pack file and its various
index files, except for the newly introduced file-based reverse index.
Now that PackReverseIndexWriter can write reverse index files,
PackWriter#writeReverseIndex will write one for a pack if the
corresponding config flag PackConfig#writeReverseIndex is on.
Change-Id: Ib75dd2bbfb9ee9366d5aacb46700d8cf8af4823a
Signed-off-by: Anna Papitto <annapapitto@google.com>
With completely independent branches, there is no merge base. In this
case, the list of commits must include the root commit of the branch to
be rebased.
Bug: 581832
Change-Id: I0f5bdf179d5b07ff09f1a274d61c7a0b1c0011c6
Signed-off-by: Thomas Wolf <twolf@apache.org>
Handle the case of the commit to be picked not having any parents.
Since JGit implements cherry-pick as a 3-way-merge between the commit
to be picked and the target commit, using the parent of the picked
commit as merge base, this is super simple: just don't set a base tree.
The merger will not find any merge base and will supply an empty tree
iterator for the base.
Bug: 581832
Change-Id: I88985f1b1723db5b35ce58bf228bc48d23d6fca3
Signed-off-by: Thomas Wolf <twolf@apache.org>
JGit's AddCommand always renormalizes tracked files. C git does so only
on git add --renormalize. Especially for git add . and the JGit
equivalent git.add().addFilepattern(".").call() this can make a big
difference if there are many files, or large files.
Add a "renormalize" option to AddCommand. To maintain compatibility with
existing uses, this option is "true" by default, and the behavior of
AddCommand is as it has always been in JGit.
If set to "false", use an IndexDiffFilter (in addition to a path filter,
if any). This skips any unchanged files (that are not racily clean) from
content checks. Note that changes in CRLF settings or in filters will be
ignored for such files if renormalize == false.
Add the "--renormalize" option to the Add command in the JGit command
line program. For the command-line program, the default is as in C git:
renormalize is off by default and enabled only if the option is given.
Note that --renormalize implies --update in the command line program, as
in C git. In AddCommand, the two settings are independent.
Additionally, avoid opening input streams unnecessarily in
WorkingTreeIterator.getEntryContentLength() and fix some bogus
indentation.
Add a simple test that adds 1000 files of 10kB in 10 directories twice
and that fails if the second invocation (without any changes) with
renormalize=false is not significantly faster.
Locally, I observe for that second invocation
* git.add().addFilepattern(".").call() ~660ms
* git.add().addFilepattern(".").setRenormalize(false).call() ~16ms
Bug: 494323
Change-Id: I30f9d518563fa55d7058a48c27c425f3b60aeb4c
Signed-off-by: Thomas Wolf <twolf@apache.org>
* stable-6.5:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
Parse pull.rebase=preserve as alias for pull.rebase=merges
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I16e8553d187a8ef541f578291f47fc39c3da4ac0
The reverse index for a pack is used to quickly find an object's
position in the pack's forward index based on that object's pack offset.
It is currently computed from the forward index by sorting the index
entries by the corresponding pack offset. This computation uses
bucket sort with insertion sort, which has an average runtime of
O(n log n) and worst case runtime of O(n^2); and memory usage of
3*size(int)*n because it maintains 3 int arrays, even after sorting is
completed. The computation must be performed every time that the reverse
index object is created in memory.
In contrast, Cgit persists a pack reverse index file to avoid
recomputing the reverse index ordering every time that it is needed.
Instead they write a file with format
https://git-scm.com/docs/pack-format#_pack_rev_files_have_the_format
which can later be read and parsed into an in-memory reverse index each
time it is needed.
Introduce these reverse index files to JGit. PackReverseIndexWriter
writes out a reverse index file to be read later when needed. Subclass
PackReverseIndexWriterV1 writes a file with the official version 1
format.
To avoid temporarily allocating an Integer collection while sorting and
writing out the contents, using memory 4*size(Integer)*n, use an
IntList and its #sort method, which uses quicksort.
Change-Id: I6437745777a16f723e2f1cfcce4e0d94e599dcee
Signed-off-by: Anna Papitto <annapapitto@google.com>
IntList is a class for working with lists of primitive ints without
boxing them into Integers. For writing the reverse index file format,
sorting ints will be needed but IntList doesn't provide a sorting
method yet.
Add the #sort method to sort an IntList by an IntComparator, using
quicksort, which has a average runtime of O(n log n) and sorts in-place
by using O(log n) stack frames for recursive calls.
Change-Id: Id69a687c8a16d46b13b28783b194a880f3f4c437
Signed-off-by: Anna Papitto <annapapitto@google.com>
* stable-6.4:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I6d20fea3a417e4361b61e81756253343668eb5de
* stable-6.3:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I463f8528e623316add204848d551c44d44d04858
* stable-6.2:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I736c7d0ed9c6e9718fa98976c3dc5a25ab8cda85
* stable-6.1:
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I9863cbce95d845efc891724898954b0b2f8dbf7b
During my development of Id7721cc5b7ea650e77c2db47042715487983cae6, I
have found this test to be flaky when run by CI. As a speculative fix,
mark this test as @Ignore so it won't be run.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Change-Id: Idfe04d7f1fb72a772d4c8d249ca86a9c2eec0b1a
The changes I1db6fcf414b and I634b92877f added tests which were failing
with errors [1] and [2] with "bazel test //...". This was not caught
because we don't have CI running with bazel. Fix bazel build file so
that these errors are no longer thrown when run with bazel.
[1] error: cannot find symbol FileRepositoryBuilderTest
[2] error: cannot find symbol RefDirectoryTest
Bug: 581816
Signed-off-by: Prudhvi Akhil Alahari <quic_prudhvi@quicinc.com>
Change-Id: I1e57111662825f5f14f373bc4f8d24cce1fec0b8
* stable-6.0:
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I5e13d5b5414aef97e518898166bfa166c692e60f
Matthias Sohn