* stable-6.1:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: I4c5b000b09287cc32f0e4d6a24a766ef4e17ddbe
* stable-6.0:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: Ia01c5ac5259b8820afb823d97bee247b5a5fb14a
* stable-5.13:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Change-Id: Id37bee59ca3c7947604c54b6d4e7c02628a657fe
* stable-5.12:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Change-Id: Ibe6652374ab5971105e62b05279f218c8c130fee
* stable-5.11:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Change-Id: I25bb99687b969f9915a7cbda8d1332bec778096a
* stable-5.10:
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Migrated "Prevent infinite loop rescanning the pack list on
PackMismatchException" to refactoring done in
https://git.eclipse.org/r/q/topic:restore-preserved-packs
Change-Id: I0fb77bb9b498d48d5da88a93486b99bf8121e3bd
* stable-5.9:
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Change-Id: I15ff2d7716ecaceb0eb87b8823d85670f5db709d
We found, when analysing an incident where Gerrit's gc runner thread got
stuck, that we can end up in an infinite loop in
ObjectDirectory#openPackedObject which tries to rescan the pack
list and starts over trying to open a packed object in an unconfined
loop if it catches a PackMismatchException.
Here the relevant part of a thread dump we created while the gc runner
was stuck:
"WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@350812a3[Not
completed,
task = java.util.concurrent.Executors$RunnableAdapter@5425d7ee]]" #72
tid=0x00007f73cee1c800 nid=0x584
runnable [0x00007f7392d57000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.jgit.internal.storage.file.WindowCache.removeAll(WindowCache.java:716)
at org.eclipse.jgit.internal.storage.file.WindowCache.purge(WindowCache.java:399)
at org.eclipse.jgit.internal.storage.file.PackFile.close(PackFile.java:296)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.reuseMap(ObjectDirectory.java:973)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacksImpl(ObjectDirectory.java:904)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacks(ObjectDirectory.java:895)
- locked <0x000000050a498f60> (a
java.util.concurrent.atomic.AtomicReference)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.searchPacksAgain(ObjectDirectory.java:794)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:465)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:417)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:408)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader$1.open(ObjectReader.java:279)
at org.eclipse.jgit.revwalk.RevWalk$2.next(RevWalk.java:1031)
at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1911)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:960)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:876)
at org.eclipse.jgit.internal.storage.file.GC.writePack(GC.java:1168)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:852)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:269)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:220)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:112)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:75)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:76)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:103)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.18/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.18/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.18/ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
The code in ObjectDirectory#openPackedObject [1] apparently assumes that
this is caused by a transient problem which it can resume from by
retrying. We use `core.trustFolderStat = false` on this server since it
uses NFS. The incident we had showed that we can enter into an infinite
loop here if there is a permanent mismatch between a pack file and its
corresponding pack index. I am not yet sure how this can happen.
Break the infinite loop by limiting the number of attempts rescanning
the pack list to 5 retries. When we exceed this threshold set the type
of the PackMismatchException to permanent and rethrow it which breaks
the infinite loop.
Also apply the same limit in #getPackedObjectSize
and #selectObjectRepresentation where we use similar retry loops.
[1] 011c26ff36/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java (465)
Change-Id: I20fb63bcc1fdc3a03d39b963f06a90e6f0ba73dc
If packing loose refs fails due to a lock failure, reject update with
a LOCK_FAILURE.
Change-Id: I100e81efd528d963231a1b87bacd9d68f9245a1b
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
Before this change, attempting to use the jgit Amazon S3 transport with an S3-compatible service that requires https (for example, Backblaze B2) results in an error:
$ jgit push b2
fatal: amazon-s3://metadaddy-jgit/repos/test/objects: error in packed-refs
This change adds a "protocol" property to the Amazon S3 transport configuration, defaulting to http, and uses that value when constructing the URL for the S3 service.
Example configuration for Backblaze B2:
accesskey: <Your B2 Application Key>
secretkey: <Your B2 Application Key Id>
acl: private
protocol: https
domain: s3.us-west-004.backblazeb2.com
region: us-west-004
aws.api.signature.version: 4
Behavior after this change:
$ jgit push b2
Counting objects: 3
Finding sources: 100% (3/3)
Getting sizes: 100% (2/2)
Compressing objects: 100% (37/37)
Writing objects: 100% (3/3)
Put pack-673f9bb.idx: 100% (1/1)
To amazon-s3://.jgit_b2@metadaddy-jgit/repos/test
* [new branch] main -> main
Change-Id: I03bdbb3510fb81a416c225a720178f42eec41b21
A commit A can reach a commit B only if the generation number of A is
strictly larger than the generation number of B. This condition allows
significantly short-circuiting commit-graph walks.
On a copy of the Linux repository where HEAD is contained in v6.3-rc4
but no earlier tag, the command 'git tag --contains HEAD' of
ListTagCommand#call() had the following peformance improvement:
(excluded the startup time of the repo)
Before: 2649ms (core.commitgraph=true)
11909ms (core.commitgraph=false)
After: 91ms (core.commitgraph=true)
11934ms (core.commitgraph=false)
Bug: 574368
Change-Id: Ia2efaa4e9ae598266f72e70eb7e3b27655cbf85b
Signed-off-by: kylezhao <kylezhao@tencent.com>
Loading of pack, bitmap and commit-graph copy the same code to adjust
the input stream buffering.
Extract to a common function. Besides reusing the code, the name hints
what it is doing.
This block aligment seems unnecessary as the reading is from storage
not dfs cache. The channel probably knows better. Left a TODO because
I don't know the original intention.
Change-Id: I18b77ae8189830fcd4d5932b6b5823b693ed6090
* stable-6.5:
Ensure parsed RevCommitCG has derived data from commit-graph
Downgrade maven-site-plugin to 3.12.1
Use wagon-ssh-external to deploy Maven site
Change-Id: Ide721fb088fa04f6276ac495968a45e732f6e139
If a RevCommitCG was newly created and called #parseCanonical(RevWalk,
byte[]) method immediately, its flag was marked as PARSED, but no
derived data was obtained from the commit-graph. This is different from
what we expected.
Change-Id: I5d417efa3c42d211f19e6acf255f761e84d84450
Signed-off-by: kylezhao <kylezhao@tencent.com>
Running
bazel test //org.eclipse.jgit.test:org_eclipse_jgit_patch_PatchApplierTest
results in a DefaultCharset error on my machine, which can be avoided
by explicitly specifying a charset when calling getBytes on a string. In
these tests, the charset used doesn't really matter, so go with UTF_8.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Change-Id: Id7721cc5b7ea650e77c2db47042715487983cae6
During my development of Id7721cc5b7ea650e77c2db47042715487983cae6, I
have found this test to be flaky when run by CI. As a speculative fix,
mark this test as @Ignore so it won't be run.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Change-Id: Idfe04d7f1fb72a772d4c8d249ca86a9c2eec0b1a
This reverts commit 9c33f7364d.
Reason for revert: This change was based on the false claim that the
packedrefs file lock is held while the CAS is being done, but it is
actually released before the CAS (the in memory lock is still held,
however that does not prevent external actors from updating the
packedrefs files and then another thread from subsequently re-reading it
and updating the in memory packedRefList). Although reverting this
change can cause the CAS to fail, it should not actually matter since
the failure would indicate that another thread has already updated the
in memory packedRefList to either the same version this thread was
trying to update it too, or to a more recent version. Either way,
failing the CAS is then appropriate and should not be problematic.
Although this change reverts the code in the RefDirectory class, it
keeps the "improvements" to the test so that it continues to pass
reliably. The reason for the quotes around the word "improvements" is
because I believe the test alteration actually dramatically changes the
intent of the test, and that the original intent of the test is
untestable with the GC and RefDirectory classes as is.
Change-Id: I3acee7527bb542996dcdfaddfb2bdb45ec444db5
Signed-off-by: Martin Fick <quic_mfick@quicinc.com>
The in-memory copy of packed refs might be outdated by the time the
packed-refs lock is acquired, so ensure the one read from disk is
used after acquiring the lock to prevent commit packed-refs from
throwing an exception. As a side-effect, since this updates the
in-memory copy of packed-refs when it is re-read from disk, it can
prevent other callers needing to re-read if it had changed.
Change-Id: I724c866b330b397e955b5eb04b259eedd9911e93
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
Since packed-refs is read from disk anyway, don't rely on the
in-memory copy as that is racy and if outdated, could result in
commit of pack-refs throwing an exception. This change also avoids
a possible unnecessary double read of packed-refs from disk.
Change-Id: I684a64991f53f8bdad58bbd248aae6522d11267d
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
There are no more callers (since Iae71cb3) of these methods that need
the returned value. These methods should not have been returning
anything in the first place as that can introduce bugs such as the
one described in Iae71cb3.
Change-Id: I1d083a91603da803a106cfb1506925a82c2ef809
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
Introduce a SnapshottingRefDirectory class which allows users to get
a snapshot of the ref database and use it in a request scope (for
example a Gerrit query) instead of having to re-read packed-refs
several times in a request.
This can potentially be further improved to avoid scanning/reading a
loose ref several times in a request. This would especially help
repeated lookups of a packed ref, where we check for the existence of
a loose ref each time.
Change-Id: I634b92877f819f8bf36a3b9586bbc1815108189a
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
Since the first attempt to read a ref is not expected to trigger
a RefsChangedEvent, update the test to ensure 'lastNotifiedModCnt'
is not 0 before we start the actual work. The test has been passing
luckily because createBareRepository in setUp() happens to bump
'lastNotifiedModCnt'.
Change-Id: Ibd981f677920e8c3b965aa742fe669c42b8c1c93
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
In the window between refs being packed (via refDb.pack) and obtaining
updates (via applyUpdates), packed-refs may have been updated by another
actor and relying on the previously read contents may lead to losing the
updates done by the other actor. To help avoid this, read packed-refs
from disk to ensure we have the latest copy after it is locked and
before committing updates to it.
Bug: 581641
Change-Id: Iae71cb30830b307d0df929c9131911ee476c711c
Signed-off-by: Kaushik Lingarkar <quic_kaushikl@quicinc.com>
This may fix the current authentication failures happening when trying
to deploy the jgit Maven site.
Change-Id: I55d4706cd041d93194af48fac9e8bfcd067e2cac
By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to reduce the size of the to-be-received packfile. If specified with negotiation tip, Git will only report commits reachable from the given tips. This is useful to speed up fetches when the user knows which local ref is likely to have commits in common with the upstream ref being fetched.
When negotation-tip is on, use the wanted refs instead of all refs as source of the "have" list to send.
This is controlled by the `fetch.usenegotationtip` flag, false by default. This works only for programmatic fetches and there is no support for it yet in the CLI.
Change-Id: I19f8fe48889bfe0ece7cdf78019b678ede5c6a32
Support the new option index.skipHash which was introduced in git 2.40
[1]. If it is set to true skip computing the git index checksum. This
accelerates Git commands that manipulate the index, such as git add, git
commit, or git status. Instead of storing the checksum, write a trailing
set of bytes with value zero, indicating that the computation was
skipped.
Accept a skipped checksum consisting of 20 null bytes when reading the
index since the option could have been set to true at the time when the
index was written.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-indexskipHash
Bug: 581723
Change-Id: I28ebe44c5ca1cbcb882438665d686452a0c111b2
Some test data introduced in [1] includes code with Apache
license. This triggers warnings in license analyzers.
Remove the licensed files and the test case as a quick relief. We
should restore the test with appropiate data in a follow-up change.
[1] https://git.eclipse.org/r/c/jgit/jgit/+/197724
Change-Id: I42670dc7d994f77d2c7f2c2156bcf1e112374022
Matthias Sohn