Support the new option index.skipHash which was introduced in git 2.40
[1]. If it is set to true skip computing the git index checksum. This
accelerates Git commands that manipulate the index, such as git add, git
commit, or git status. Instead of storing the checksum, write a trailing
set of bytes with value zero, indicating that the computation was
skipped.
Accept a skipped checksum consisting of 20 null bytes when reading the
index since the option could have been set to true at the time when the
index was written.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-indexskipHash
Bug: 581723
Change-Id: I28ebe44c5ca1cbcb882438665d686452a0c111b2
As of today, we don't have test coverage for RefDirectory when
core.trustPackedRefsStat config is set to after_open. Thus create new
test classes which set core.trustPackedRefsStat config to after_open in
setup and extend RefDirectoryTest and FileRepositoryBuilderTest
respectively.
Change-Id: I1db6fcf414bc488106ad4c85fb934480f299c995
Signed-off-by: Prudhvi Akhil Alahari <quic_prudhvi@quicinc.com>
* stable-6.3:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Change-Id: I5af34c92e423a651db53b4dc45ed844d5f39910d
* stable-6.2:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Change-Id: I5b6b10622b61fde3f0f10455a74ae159a0b69082
* stable-6.1:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Change-Id: I3ffe92566cc145053bb762f612dd96bc6d542c62
* stable-6.0:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Change-Id: Idea23e555c024557d7e39a86efe25f609400b962
* stable-5.13:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Change-Id: I708d0936fa86b028e4da4e7e21f332f8b48ad293
* stable-6.3:
Shortcut during git fetch for avoiding looping through all local refs
FetchCommand: fix fetchSubmodules to work on a Ref to a blob
Silence API warnings introduced by I466dcde6
Allow the exclusions of refs prefixes from bitmap
PackWriterBitmapPreparer: do not include annotated tags in bitmap
BatchingProgressMonitor: avoid int overflow when computing percentage
Speedup GC listing objects referenced from reflogs
FileSnapshotTest: Add more MISSING_FILE coverage
Change-Id: Iefcf5d832bd0087c1027876f2200689e1150abce
* stable-6.2:
Shortcut during git fetch for avoiding looping through all local refs
FetchCommand: fix fetchSubmodules to work on a Ref to a blob
Silence API warnings introduced by I466dcde6
Allow the exclusions of refs prefixes from bitmap
PackWriterBitmapPreparer: do not include annotated tags in bitmap
BatchingProgressMonitor: avoid int overflow when computing percentage
Speedup GC listing objects referenced from reflogs
FileSnapshotTest: Add more MISSING_FILE coverage
Change-Id: I2ff386d9a096277360e6c7bd5535b49984620fb3
* stable-6.1:
Shortcut during git fetch for avoiding looping through all local refs
FetchCommand: fix fetchSubmodules to work on a Ref to a blob
Silence API warnings introduced by I466dcde6
Allow the exclusions of refs prefixes from bitmap
PackWriterBitmapPreparer: do not include annotated tags in bitmap
BatchingProgressMonitor: avoid int overflow when computing percentage
Speedup GC listing objects referenced from reflogs
FileSnapshotTest: Add more MISSING_FILE coverage
Change-Id: Iff2fba026b49463016015b2fae1a42cf76ee2dbb
* stable-6.0:
Shortcut during git fetch for avoiding looping through all local refs
FetchCommand: fix fetchSubmodules to work on a Ref to a blob
Silence API warnings introduced by I466dcde6
Allow the exclusions of refs prefixes from bitmap
PackWriterBitmapPreparer: do not include annotated tags in bitmap
BatchingProgressMonitor: avoid int overflow when computing percentage
Speedup GC listing objects referenced from reflogs
FileSnapshotTest: Add more MISSING_FILE coverage
Change-Id: Ib5055f2f3b8a313c178d6f6c7c5630285ad5a726
* stable-5.13:
Shortcut during git fetch for avoiding looping through all local refs
FetchCommand: fix fetchSubmodules to work on a Ref to a blob
Silence API warnings introduced by I466dcde6
Allow the exclusions of refs prefixes from bitmap
PackWriterBitmapPreparer: do not include annotated tags in bitmap
BatchingProgressMonitor: avoid int overflow when computing percentage
Speedup GC listing objects referenced from reflogs
FileSnapshotTest: Add more MISSING_FILE coverage
Change-Id: I58ad4c210a5e7e5a1ba6b22315b04211c8909950
When running a GC.repack() against a repository with over one
thousands of refs/heads and tens of millions of ObjectIds,
the calculation of all bitmaps associated with all the refs
would result in an unreasonable big file that would take up to
several hours to compute.
Test scenario: repo with 2500 heads / 10M obj Intel Xeon E5-2680 2.5GHz
Before this change: 20 mins
After this change and 2300 heads excluded: 10 mins (90s for bitmap)
Having such a large bitmap file is also slow in the runtime
processing and have negligible or even negative benefits, because
the time lost in reading and decompressing the bitmap in memory
would not be compensated by the time saved by using it.
It is key to preserve the bitmaps for those refs that are mostly
used in clone/fetch and give the ability to exlude some refs
prefixes that are known to be less frequently accessed, even
though they may actually be actively written.
Example: Gerrit sandbox branches may even be actively
used and selected automatically because its commits are very
recent, however, they may bloat the bitmap, making it ineffective.
A mono-repo with tens of thousands of developers may have
a relatively small number of active branches where the
CI/CD jobs are continuously fetching/cloning the code. However,
because Gerrit allows the use of sandbox branches, the
total number of refs/heads may be even tens to hundred
thousands.
Change-Id: I466dcde69fa008e7f7785735c977f6e150e3b644
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
The annotated tags should be excluded from the bitmap associated
with the heads-only packfile. However, this was not happening
because of the check of exclusion of the peeled object instead
of the objectId to be excluded from the bitmap.
Sample use-case:
refs/heads/main
^
|
commit1 <-- commit2 <- annotated-tag1 <- tag1
^
|
commit0
When creating a bitmap for the above commit graph, before this
change all the commits are included (3 bitmaps), which is
incorrect, because all commits reachable from annotated tags
should not be included.
The heads-only bitmap should include only commit0 and commit1
but because PackWriterBitPreparer was checking for the peeled
pointer of tag1 to be excluded (commit2) which was not found in
the list of tags to exclude (annotated-tag1), the commit2 was
included, even if it wasn't reachable only from the head.
Add an additional check for exclusion of the original objectId
for allowing the exclusion of annotated tags and their pointed
commits. Add one specific test associated with an annotated tag
for making sure that this use-case is covered also.
Example repository benchmark for measuring the improvement:
# refs: 400k (2k heads, 88k tags, 310k changes)
# objects: 11M (88k of them are annotate tags)
# packfiles: 2.7G
Before this change:
GC time: 5h
clone --bare time: 7 mins
After this change:
GC time: 20 mins
clone --bare time: 3 mins
Bug: 581267
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Change-Id: Iff2bfc6587153001837220189a120ead9ac649dc
When cloning huge repositories I observed percentage of object counts
turning negative. This happened if lastWork * 100 exceeded
Integer.MAX_VALUE.
Change-Id: Ic5f5cf5a911a91338267aace4daba4b873ab3900
Add a couple tests that confirm what the docs say about isModified() and
equals(MISSING_FILE) behavior.
Change-Id: I6093040ba3594934c3270331405a44b2634b97c5
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
The NameConflictTreeWalk class is used in merge for iterating over
entries in commits. The class uses a separate iterator for each
commit's tree. In rare cases it can incorrectly report the same entry
twice. As a result, duplicated entries are added to the merge result
and later jgit throws an exception when it tries to process merge
result.
The problem appears only when there is a directory-file conflict for
the last item in trees. Example from the bug:
Commit 1:
* subtree - file
* subtree-0 - file
Commit 2:
* subtree - directory
* subtree-0 - file
Here the names are ordered like this:
"subtree" file <"subtree-0" file < "subtree" directory.
The NameConflictTreeWalk handles similar cases correctly if there are
other files after subtree... in commits - this is processed in the
AbstractTreeIterator.min function. Existing code has a special
optimization for the case, when all trees are pointed to the same
entry name - it skips additional checks. However, this optimization
incorrectly skips checks if one of trees reached the end.
The fix processes a situation when some trees reached the end, while
others are still point to an entry.
bug: 535919
Change-Id: I62fde3dd89779fac282479c093400448b4ac5c86
Keys used for identifying reverse indexes in the DfsBlockCache use a
custom subclass ForReverseIndex because there was no PackExt for them.
This conflates BlockCacheMetrics for reverse indexes with those for
packs, since the key falls back onto 0 when there is no extension.
Replace the custom ForReverseIndex with a DfsStreamKey usage to bring
keys for the new REVERSE_INDEX extension in line with INDEX and BITMAP
and separate reverse index and pack BlockCacheMetrics.
Change-Id: I305e2c16d2a8cb2a824855ea92e0c9a9b188fce5
Signed-off-by: Anna Papitto <annapapitto@google.com>
Make sure we always get consistent results, whether or not we have the
full data in the buffer.
Change-Id: Ieb379a0c375ad3dd352e63ac2f23bda6ef16c215
Signed-off-by: Thomas Wolf <twolf@apache.org>
* stable-6.3:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Fix API breakage caused by extracting WorkTreeUpdater
Extract Exception -> HTTP status code mapping for reuse
Don't handle internal git errors as an HTTP error
Ignore IllegalStateException if JVM is already shutting down
Allow to perform PackedBatchRefUpdate without locking loose refs
Change-Id: Ib58879be292c54a2a7f4936ac0986997985c822b
* stable-6.2:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Change-Id: I9c1576011c11b4ff8f453d18d9e786cee59860fa
* stable-6.1:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Change-Id: Ie433c46a01a0f33848d54ecf99b30a44ca01e286
* stable-6.0:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Change-Id: I176419026c3f4fdd8ebd34c61468c1ec3482ff45
* stable-5.13:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Change-Id: I40105336f0b9e593a8a2c242a9557f854c274fdc
The change If6da9833 moved the computation of SHA1 from the JVM's
JCE to a pure Java implementation with collision detection.
The extra security for public sites comes with a cost of slower
SHA1 processing compared to the native implementation in the JDK.
When JGit is used internally and not exposed to any traffic from
external or untrusted users, the extra cost of the pure Java SHA1
implementation can be avoided, falling back to the previous
native MessageDigest implementation.
Bug: 580310
Change-Id: Ic24c0ba1cb0fb6282b8ca3025ffbffa84035565e
and update
- com.sun.jna to 5.12.1.v20221103-2317
- com.sun.jna.platform to 5.12.1.v20221103-2317
- org.bouncycastle.bcpg to 1.72.0.v20221013-1810
- org.bouncycastle.bcpkix to 1.72.0.v20221013-1810
- org.bouncycastle.bcprov to 1.72.0.v20221013-1810
- org.bouncycastle.bcutil to 1.72.0.v20221013-1810
- org.mockito.mockito-core to 4.8.1.v20221103-2317
- org.objenesis to 3.3.0.v20221103-2317
Change-Id: If00094d23e51d5f66928f83c1334aa6b18b98dfe
Before this change JGit did not support the session-id capability
implemented by native Git in UploadPack. This change implements
advertising the capability from the server and parsing the session-id
received from the client during an UploadPack operation.
Enable the transfer.advertisesid config setting to advertise the
capability from the server. The client may send a session-id capability
in response. If received, the value from this is parsed and available
via the getClientSID method on the UploadPack object.
This change does not add the capability to send a session-id from the
JGit client.
https://git-scm.com/docs/gitprotocol-capabilities#_session_idsession_id
Change-Id: Ib1b6929ff1b3a4528e767925b5e5c44b5d18182f
Signed-off-by: Josh Brown <sjoshbrown@google.com>
The config setting to enable advertising the session-id capability is
currently read in the ReceivePack class. This change moves it to a
common location in the TransferConfig class so that it can be reused
in other places like UploadPack. TransferConfig is also a more logical
place for the setting as it resides in the `transfer` config section.
Set the transfer.advertisesid setting to true to send the session-id
capability to the client.
Change-Id: If68ecb5e68b59f5c452a7992d02e3688b0a86747
Signed-off-by: Josh Brown <sjoshbrown@google.com>
In protocol V0 the client capabilities are appended to the first line.
Parsing session-id is currently only supported during a ReceivePack
operation. This change will parse the client session-id capability if
it has been sent by the client.
If the server sends the session-id capability to the client. The client
may respond with a session ID of its own. FirstWant.fromLine will now
parse the ID and make it available via the getClientSID method.
This change does not add support to send the session-id capability from
the server. The change is necessary to support session-id in UploadPack.
Change-Id: Id3fe44fdf9a72984ee3de9cf40cc4e71d434df4a
Signed-off-by: Josh Brown <sjoshbrown@google.com>
Before this change JGit did not support the session-id capability
implemented by native Git. This change implements advertising the
capability from the server and parsing the session-id received from
the client during a ReceivePack operation.
Enable the transfer.advertisesid config setting to advertise the
capability from the server. The client may send a session-id capability
in response. If received, the value from this is parsed and available
via the getClientSID method on the ReceivePack object. All capabilities
in the form `capability=value` are now split into key value pairs at the
first `=` character. This change replaces specific handling for the
agent capability.
This change does not add advertisement or parsing to UploadPack. This
change also does not add the ability to send a session ID from the JGit
client.
https://git-scm.com/docs/protocol-v2/2.33.0#_session_idsession_id
Change-Id: I56fb115e843b11b27e128c4ac427b05d5ec129d0
Signed-off-by: Josh Brown <sjoshbrown@google.com>
Reachability bitmaps are designed to speed up the "counting objects"
phase of generating a pack during a clone or fetch. They are not
optimized for Git clients sending a small topic branch via "git push".
In some cases (see [1]), using reachability bitmaps during "git push"
can cause significant performance regressions.
Add PushCommand#setUseBitmaps(boolean) to allow users to tell "git push"
not to use bitmaps.
[1]: https://lore.kernel.org/git/87zhoz8b9o.fsf@evledraar.gmail.com/
Change-Id: I7fb7d26084ec63ddfa7249cf58abb85929b30e56
Signed-off-by: kylezhao <kylezhao@tencent.com>
Fix and complete the implementation of calling the pre-push hook.
Add the missing error stream redirect, and add the missing setters
in Transport and in PushCommand. In Transport, delay setting up a
PrePushHook such that it happens only on a push. Previously, the
hook was set up also for fetches.
Bug: 549246
Change-Id: I64a576dfc6b139426f05d9ea6654027ab805734e
Signed-off-by: Thomas Wolf <twolf@apache.org>
We need the full size of the objects to populate the object-size index
of a pack. This size is not always the one encoded in the object header
in the pack (e.g. for deltas).
Populate the full size of PackedObjectInfos in the PackParser, which is
invoked when receiving a pack e.g. in a push.
Change-Id: I102c20901aefb5e85047e2e526c0d733f82ff74b
Matthias Sohn