In findGraphPosition, when there is no object whose OID starts with
the first byte of the sought OID, low equals high. This violates an
invariant of the loop, and when the sought OID is lexicographically
greater than every other OID in the repository, causes an
ArrayIndexOutOfBoundsException (because we're trying to read outside the
list of OIDs).
Therefore, check the "low < high" condition at the start of the loop,
not only after the first iteration.
Change-Id: Ic8ac198c151bd161c4996b9e7cb6e6660f151733
Helped-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
If a method called in a finally block throws an exception we should add
exceptions caught earlier to the exception we throw in the finally block
not regarding if it's a checked or unchecked exception.
Change-Id: I4c6be9a3a08482b07659ca31d6987ce719d81ca5
Errorprone raises the following warning:
"[ReferenceEquality] Comparison using reference equality
instead of value equality".
Change-Id: Iacb207ef0625bb987a08406d4e7461e48fade97f
New static final constant is a (very minor) API break that needs to be
suppressed explicitly despite @since 6.6.
Remove a number of no longer needed API filters, and fix a broken
$NON-NLS-1$.
Change-Id: Ie4b0c45e8bd1f3067b6ff81c07d4b21b50bb8685
Signed-off-by: Thomas Wolf <twolf@apache.org>
During garbage collection, extensions for temporary files for indices
are formatted manually.
Add a method to PackExt to generate the temporary file extensions for
each type of index file programmatically.
Change-Id: I210bc2702e750bf0aea643b1a9a8536adebef179
Signed-off-by: Anna Papitto <annapapitto@google.com>
ServiceV2 is not collecting wants/have in PackStatistics. This records
the stats for fetch and push-negotiation.
Change-Id: Iefd79f36b3d7837195e8bd9fc7007de352089e66
PackWriter offers the ability to write out the pack file and its various
index files, except for the newly introduced file-based reverse index.
Now that PackReverseIndexWriter can write reverse index files,
PackWriter#writeReverseIndex will write one for a pack if the
corresponding config flag PackConfig#writeReverseIndex is on.
Change-Id: Ib75dd2bbfb9ee9366d5aacb46700d8cf8af4823a
Signed-off-by: Anna Papitto <annapapitto@google.com>
The in process lock is intended to manage contention on locking the
packed-refs file within a single process without acquiring the file
system lock. Not sharing it across RefDirectory instances of the same
repository undermines that intent and results in more contention at the
file system level.
Change-Id: I68f11856aa0b4b1524f43554d7391a322a0a6897
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
With completely independent branches, there is no merge base. In this
case, the list of commits must include the root commit of the branch to
be rebased.
Bug: 581832
Change-Id: I0f5bdf179d5b07ff09f1a274d61c7a0b1c0011c6
Signed-off-by: Thomas Wolf <twolf@apache.org>
Handle the case of the commit to be picked not having any parents.
Since JGit implements cherry-pick as a 3-way-merge between the commit
to be picked and the target commit, using the parent of the picked
commit as merge base, this is super simple: just don't set a base tree.
The merger will not find any merge base and will supply an empty tree
iterator for the base.
Bug: 581832
Change-Id: I88985f1b1723db5b35ce58bf228bc48d23d6fca3
Signed-off-by: Thomas Wolf <twolf@apache.org>
JGit's AddCommand always renormalizes tracked files. C git does so only
on git add --renormalize. Especially for git add . and the JGit
equivalent git.add().addFilepattern(".").call() this can make a big
difference if there are many files, or large files.
Add a "renormalize" option to AddCommand. To maintain compatibility with
existing uses, this option is "true" by default, and the behavior of
AddCommand is as it has always been in JGit.
If set to "false", use an IndexDiffFilter (in addition to a path filter,
if any). This skips any unchanged files (that are not racily clean) from
content checks. Note that changes in CRLF settings or in filters will be
ignored for such files if renormalize == false.
Add the "--renormalize" option to the Add command in the JGit command
line program. For the command-line program, the default is as in C git:
renormalize is off by default and enabled only if the option is given.
Note that --renormalize implies --update in the command line program, as
in C git. In AddCommand, the two settings are independent.
Additionally, avoid opening input streams unnecessarily in
WorkingTreeIterator.getEntryContentLength() and fix some bogus
indentation.
Add a simple test that adds 1000 files of 10kB in 10 directories twice
and that fails if the second invocation (without any changes) with
renormalize=false is not significantly faster.
Locally, I observe for that second invocation
* git.add().addFilepattern(".").call() ~660ms
* git.add().addFilepattern(".").setRenormalize(false).call() ~16ms
Bug: 494323
Change-Id: I30f9d518563fa55d7058a48c27c425f3b60aeb4c
Signed-off-by: Thomas Wolf <twolf@apache.org>
* stable-6.5:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
Parse pull.rebase=preserve as alias for pull.rebase=merges
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I16e8553d187a8ef541f578291f47fc39c3da4ac0
The reverse index for a pack is used to quickly find an object's
position in the pack's forward index based on that object's pack offset.
It is currently computed from the forward index by sorting the index
entries by the corresponding pack offset. This computation uses
bucket sort with insertion sort, which has an average runtime of
O(n log n) and worst case runtime of O(n^2); and memory usage of
3*size(int)*n because it maintains 3 int arrays, even after sorting is
completed. The computation must be performed every time that the reverse
index object is created in memory.
In contrast, Cgit persists a pack reverse index file to avoid
recomputing the reverse index ordering every time that it is needed.
Instead they write a file with format
https://git-scm.com/docs/pack-format#_pack_rev_files_have_the_format
which can later be read and parsed into an in-memory reverse index each
time it is needed.
Introduce these reverse index files to JGit. PackReverseIndexWriter
writes out a reverse index file to be read later when needed. Subclass
PackReverseIndexWriterV1 writes a file with the official version 1
format.
To avoid temporarily allocating an Integer collection while sorting and
writing out the contents, using memory 4*size(Integer)*n, use an
IntList and its #sort method, which uses quicksort.
Change-Id: I6437745777a16f723e2f1cfcce4e0d94e599dcee
Signed-off-by: Anna Papitto <annapapitto@google.com>
IntList is a class for working with lists of primitive ints without
boxing them into Integers. For writing the reverse index file format,
sorting ints will be needed but IntList doesn't provide a sorting
method yet.
Add the #sort method to sort an IntList by an IntComparator, using
quicksort, which has a average runtime of O(n log n) and sorts in-place
by using O(log n) stack frames for recursive calls.
Change-Id: Id69a687c8a16d46b13b28783b194a880f3f4c437
Signed-off-by: Anna Papitto <annapapitto@google.com>
* stable-6.4:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I6d20fea3a417e4361b61e81756253343668eb5de
* stable-6.3:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I463f8528e623316add204848d551c44d44d04858
* stable-6.2:
[bazel] Move ToolTestCase to src folder (6.2)
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I736c7d0ed9c6e9718fa98976c3dc5a25ab8cda85
* stable-6.1:
GcConcurrentTest: @Ignore flaky testInterruptGc
Fix CommitTemplateConfigTest
Fix after_open config and Snapshotting RefDir tests to work with bazel
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I9863cbce95d845efc891724898954b0b2f8dbf7b
* stable-6.0:
[bazel] Skip ConfigTest#testCommitTemplatePathInHomeDirecory
Demote severity of some error prone bug patterns to warnings
UploadPack: Fix NPE when traversing a tag chain
Change-Id: I5e13d5b5414aef97e518898166bfa166c692e60f
This ensures backwards compatibility to the old config value which was
removed in git 2.34 which JGit followed in Ic07ff954e2.
Change-Id: I2b4e27fd71898b6e0e227e406c40682bd9786cd4
Always parse RevTags including their body before getting their object
to ensure that non-cached objects are handled correctly when traversing
a tag chain. An NPE in UploadPack#addTagChain will occur on a depth=1
fetch of a branch containing a tag chain and the ref to one of the
middle tags in the chain is deleted.
Change-Id: Ifd8fe868869070b365df926fec5dcd8e64d4f521
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
* stable-6.5:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: I0b03ca566053a158c6c8e75ccec8360a2f368ed9
* stable-6.4:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Change-Id: I89af76946014fb44bd64c20e2b01a53397768d90
* stable-6.3:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: I18b46be0f09535c61efabe24ab1579faa3d06ba8
* stable-6.2:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: I8006068f16ae442a2246e043a680053f2af34e9f
* stable-6.1:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: I4c5b000b09287cc32f0e4d6a24a766ef4e17ddbe
* stable-6.0:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: Ia01c5ac5259b8820afb823d97bee247b5a5fb14a
* stable-5.13:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Change-Id: Id37bee59ca3c7947604c54b6d4e7c02628a657fe
* stable-5.12:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Change-Id: Ibe6652374ab5971105e62b05279f218c8c130fee
* stable-5.11:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: I25bb99687b969f9915a7cbda8d1332bec778096a
* stable-5.10:
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Migrated "Prevent infinite loop rescanning the pack list on
PackMismatchException" to refactoring done in
https://git.eclipse.org/r/q/topic:restore-preserved-packs
Change-Id: I0fb77bb9b498d48d5da88a93486b99bf8121e3bd
* stable-5.9:
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Change-Id: I15ff2d7716ecaceb0eb87b8823d85670f5db709d
We found, when analysing an incident where Gerrit's gc runner thread got
stuck, that we can end up in an infinite loop in
ObjectDirectory#openPackedObject which tries to rescan the pack
list and starts over trying to open a packed object in an unconfined
loop if it catches a PackMismatchException.
Here the relevant part of a thread dump we created while the gc runner
was stuck:
"WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@350812a3[Not
completed,
task = java.util.concurrent.Executors$RunnableAdapter@5425d7ee]]" #72
tid=0x00007f73cee1c800 nid=0x584
runnable [0x00007f7392d57000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.jgit.internal.storage.file.WindowCache.removeAll(WindowCache.java:716)
at org.eclipse.jgit.internal.storage.file.WindowCache.purge(WindowCache.java:399)
at org.eclipse.jgit.internal.storage.file.PackFile.close(PackFile.java:296)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.reuseMap(ObjectDirectory.java:973)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacksImpl(ObjectDirectory.java:904)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacks(ObjectDirectory.java:895)
- locked <0x000000050a498f60> (a
java.util.concurrent.atomic.AtomicReference)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.searchPacksAgain(ObjectDirectory.java:794)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:465)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:417)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:408)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader$1.open(ObjectReader.java:279)
at org.eclipse.jgit.revwalk.RevWalk$2.next(RevWalk.java:1031)
at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1911)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:960)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:876)
at org.eclipse.jgit.internal.storage.file.GC.writePack(GC.java:1168)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:852)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:269)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:220)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:112)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:75)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:76)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:103)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.18/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.18/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.18/ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
The code in ObjectDirectory#openPackedObject [1] apparently assumes that
this is caused by a transient problem which it can resume from by
retrying. We use `core.trustFolderStat = false` on this server since it
uses NFS. The incident we had showed that we can enter into an infinite
loop here if there is a permanent mismatch between a pack file and its
corresponding pack index. I am not yet sure how this can happen.
Break the infinite loop by limiting the number of attempts rescanning
the pack list to 5 retries. When we exceed this threshold set the type
of the PackMismatchException to permanent and rethrow it which breaks
the infinite loop.
Also apply the same limit in #getPackedObjectSize
and #selectObjectRepresentation where we use similar retry loops.
[1] 011c26ff36/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java (465)
Change-Id: I20fb63bcc1fdc3a03d39b963f06a90e6f0ba73dc
If packing loose refs fails due to a lock failure, reject update with
a LOCK_FAILURE.
Change-Id: I100e81efd528d963231a1b87bacd9d68f9245a1b
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
Matthias Sohn