* stable-6.6:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I29241619e6c09933bb856e486f379be10dd609c2
* stable-6.5:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I7272a22451c0de6b4770767e7bb4e24c81518c20
* stable-6.4:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I2951d01f5f4581bee20079508cd8ee6ca8554f1f
* stable-6.3:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I4b94a2b79941c085fa2f62246e8e879aaa85cd3f
* stable-6.2:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I22b89bf00dcef26b2096d25397aa9a57a745a92b
* stable-6.1:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: Ib4e4fe407dce334c7537bf278baa39db93aa2f09
* stable-6.0:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: I0c9c0b3c206cac03a93b30eda348177a4de35c36
* stable-5.13:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Change-Id: Ifeaa4b4f0c5944d4ecd3042be429833ff72b43ed
Change Ide3445e652 introduced the `--pack-kept-objects` option to GC for
including the objects contained in the locked packfiles during the
repack phase.
Whilst this allowed to explicitly pass a command line argument to the
jgit gc program, it did not allow the option to be read from
configuration.
Allow the pack kept objects option to be configured exactly as C-Git
documents [1], by introducing a new `repack.packKeptObjects`
configuration.
`repack.packKeptObjects` defaults to `true`, when the
`pack.buildBitmaps` is `true` (which is the default case), `false`
otherwise.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-repackpackKeptObjects
Bug: 582292
Change-Id: Ia931667277410d71bc079d27c097a57094299840
Packfiles having an equivalent .keep file are associated with in-flight
pushes that haven't been completed, with potentially a set of git
objects not yet referenced by a ref.
If the Git client is not up-to-date, it may result in pushing a
packfile, generating a <packfile>.keep on the server, which
may also contain existing commits due to the lack of Git protocol
negotiation in the git-receive-pack.
The Git protocol negotiation is the phase where the client and the
server exchange the list of refs they have for trying to find a common
base and minimise the amount of objects to be transferred.
The repack phase in GC was previously skipping all objects that were
contained in all packfiles having a <packfile>.keep file associated
(aka "locked packfiles"), which did not take into consideration the
fact that excluding the existing commits would have resulted in the
generation of an invalid bitmap file.
The code for excluding the objects in the locked packfiles was written
well before the bitmap was introduced, hence could not consider a use
case that did not exist at that time.
However, when the bitmap was introduced, the exclusion of locked
packfiles was not changed, hence creating a potential problem.
The issue went unnoticed for many years because the bitmap generation
was disabled when JGit noticed any locked packfiles; however, the
bitmaps are enabled again since Id722e68d9f , and the the issue is now
visible and is impacting the GC repack phase.
Introduce the '--pack-kept-objects' option in GC for including the
objects contained in the locked packfiles during the repack phase,
which is not an issue because of the following:
- If there are any existing commits duplicated in the packfiles
they will be just considered once anyway because the repack doesn't
generate duplicates in the output packfile.
- If there are any new commits that do not have any ref pointing to
them, they will be automatically excluded from the output repacked
packfile.
The same identical solution is adopted in the C implementation of git
in repack.c.
Because the locked packfile is not pruned, any new commits not pointed
by any refs will remain in the repository and there will not be any
accidental pruning or object loss as it is today before this change.
As a side-effect of this change, it is now potentially possible to still
have duplicate BLOBs after GC when the keep packfile contained existing
objects. However, it is way better to keep the duplication until the
next GC phase rather than omitting existing objects from repacking and,
therefore generating an invalid bitmap and incorrect packfile.
Bug: 582292
Bug: 582455
Change-Id: Ide3445e652fcf256a7912f881cb898897c99b8f8
When checking out a file into the working tree ensure that all parent
directories of the file below the working tree root are actually
directories and do exist before we try to create the file.
When multiple files are to be checked out (or even a whole tree), this
may check the same directories over and over again. Asking the file
system every time for file attributes is a potentially expensive
operation. As a remedy, introduce an in-memory cache of directory
states for a particular check-out operation.
Apply the same fix also in the ResolveMerger, which may also check out
files, and also in the PatchApplier. In PatchApplier, also validate
paths.
Change-Id: Ie12864c54c9f901a2ccee7caddec73027f353111
Signed-off-by: Thomas Wolf <twolf@apache.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
CommitGraphWriter first defines the chunks and then writes them. If at
write time a chunk is unknown, it is ignored. This is brittle: if
somebody adds a chunk to the header but not to the actual writing, the
commit-graph is broken and there is no error reported anywhere.
Throw exception if at write time a chunk is unknown. This can only
happen by a coding error in the writer.
Change-Id: Iade677bb6ce368b6941b75a21c622917afa3b751
C git uses this alternate fallback location if the file exists and
~/.gitconfig does not. Implement this also for JGit.
If both files exist, reading behavior is as if the XDG config was
inserted between the HOME config and the system config. Writing
behaviour is different: all changes will be applied only in the HOME
config. Updates will occur in the XDG config only if the HOME config
does not exist.
This is consistent with the behavior of C git; compare [1], especially
the sections on FILES and SCOPES, and the description of the --global
option.
[1] https://git-scm.com/docs/git-config
Bug: 581875
Change-Id: I2460b9aa963fd2811ed8a5b77b05107d916f2b44
Signed-off-by: Thomas Wolf <twolf@apache.org>
The benefit is that certain InputStreams can override the default
implementation for performance reasons.
Change-Id: I4c924157ec0f0ec63b0eca7cdbdc9325af24cab6
Introduce a core.packedIndexGitUseStrongRefs configuration key, which
defaults to true so that the current behavior does not change. However,
setting it to false allows soft references to be used for Pack indices
instead of strong references so that they can be garbage collected when
there is memory pressure.
Pack objects can be large when associated with pack files with large
object counts, and this memory is not really accounted for or tracked by
the WindowCache and it can be very substantial at times, especially with
many large object count projects. A particularly problematic use case is
Gerrit's ls-projects command which loads very little data in the
WindowCache via ByteWindows, but ends up loading and holding many entire
indices in memory, sometimes even after the ByteWindows for their Pack
objects have already been garbage collected since they won't get cleared
until after a new ByteWindow is loaded. By using SoftReferences, single
use indices can get cleared when there is memory pressure and OOMs can
be easily avoided, drastically reducing the amount of memory required to
perform an ls-projects on large sites with many projects and large
object counts.
On one of our test sites, an ls-projects command with strong index
references requires more than 66GB of heap to complete successfully,
with soft index references it requires less than 23GB.
Change-Id: I3cb3df52f4ce1b8c554d378807218f199077d80b
Signed-off-by: Martin Fick <quic_mfick@quicinc.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
A subclass cannot implement a listener with the default access.
Make the interface protected. Not public because so far only
subclasses are interested in this interface. We can widen the
visibility later if needed.
Change-Id: I54e5c0ef1312dfe2fa660bc8fb54e2be35c0f6df
Provide commit graph stats in the same way that we provide reftable
stats.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Change-Id: Ib80c892a26f9b552bc90f3cbe7da83b02ffebdfd
GC puts all commits reachable from heads and tags into the GC pack,
and commits reachable only from other refs (e.g. refs/changes) into
GC_REST. The commit-graph contains all commits in GC and GC_REST. This
produces too big commit graphs in some repos, beating the purpose of
loading the index.
Limit the commit graph to commits reachable from heads and tags
(i.e. commits in the GC pack).
Change-Id: I4962faea5a726d2ea3e548af0aeae370a6cc8588
We want to measure the data used to serve a request. As a first step,
we want to know how many indices are accessed during the request and
their sizes.
Expose an interface in DfsReader to announce when an index is loaded
into the reader, i.e. when its reference is set.
The interface is more flexible to implementors (what/how to collect)
than the existing DfsReaderIOStats object.
Change-Id: I56f7658fde1758efaf869fa779d11b533a81a0a7
* stable-6.6:
Update to Tycho 4.0.1
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I7294c21748897eb3f94eeffbda944b62e3206c0d
* stable-6.5:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: Id2e49252a9dc268210c9439848e77604885371aa
* stable-6.4:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: Idb6dd6160e023673e3650653a15f6b1c540de96e
* stable-6.3:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I0bccc36d9cc9a36f1be9b1562df35ce3a0e95eee
* stable-6.2:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I589ed444b5cbfc5b073cac91323e2cc97ab98087
* stable-6.1:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I5b16c3b613a95b7f28c8f6ac0b20c4c593759cea
* stable-6.0:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: Ib08037f6055dac1776e38cfb4ff8c88a50ad3e60
* stable-5.13:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Change-Id: I1f50995d9d9c592ec0e02a04e0e409440b49f9f3
This allows extracting footers from a messages not associated with a
commit.
The public API of RevCommit is kept intact.
Change-Id: I5809c23df7b7d49641a4be3a26d6f987d3d57c9b
Bug: Google b/287891316
RevCommit and RevCommitCG were designed like "pointers" to data that
load the content on demand, not on construction. This saves memory.
Make the loading of changed-path-filter follow the same pattern. The
ChangedPathFilters are only pointers to locations in the commit-graph
(not the actual data), so the memory saving is not that big, but this
is more consistent with the rest of the API.
As 6.7 is not released, we can still change the RevWalk API.
Change-Id: Id4186ea744b8a2418d0329facae69f785108d356
Matthias Sohn