Log as warning when an attempt to remove a directory
fails. This helps troubleshooting some bugs like the GC leaving
behind empty directories.
Change-Id: Idb94ce17f8be9668a970c7ecae31436bf434073c
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
* stable-4.10:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: I2f02298d0ff7caafeca4020cde4fdfa29a46e585
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.9:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: Ibd8f2a041b0de6e008a1ea84b92823f8cbc6e3d2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.8:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: Id6a85804695d5dcb32f26ed1d861b7c93577c5e4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.7:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: If17328fbd101d596a8a16d9c4a190e9b6e120902
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This change fixes the issue [1]. Before this fix, a merge involving
the caching of consecutive yet similar filenames with Norwegian
characters [2] used to throw an IllegalStateException: Duplicate
stages not allowed. This was caused by inaccurate decoding of the
filenames, using string values assuming default encoding. In the
toString method of DirCacheEntry, used before through getPathString,
UTF-8 encoding is used, but the end result becomes default encoding,
through Object's default toString usage. The special characters in
those two consecutive (particular) filenames [2] were becoming the
very same decoded /single character, lending consecutive -but then
identical- filenames. Thus the perceived duplicate 0-staging of the
file(s).
Replace getPathString usage with getRawPath for this specific case,
or use byte array representations of cached entries instead of string.
Adding a test for this change is not possible, as there is no known
way to change the default encoding for filenames such as [2] (e.g.).
JGitTestUtil does write file contents through UTF-8, but encoding like
so does not apply to the actual file name. Hence there is no way to
create files with names properly made of special characters such as
[2]'s. And the test that is necessary for this case assumes such
Norwegian (or similar characters) filenames. Changing the default
locale programmatically in a test has no effect either. And changing
the LANG value passed to the JVM is only possible upon starting it.
[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=9153
[2] <=>
(...)
"a/b/SíÒr-Norge.map",
"a/b/Sør-Norge.map",
(...)
Change-Id: Ib9f2f5297932337c9817064cc09d9f774dd168f4
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
From the javadoc for Files.list:
"The returned stream encapsulates a DirectoryStream. If timely disposal
of file system resources is required, the try-with-resources construct
should be used to ensure that the stream's close method is invoked
after the stream operations are completed."
This is the only call to Files#newDirectoryStream that is not already in
a try-with-resources.
Change-Id: I91e6c56b5d74e8435457ad6ed9e6b4b24d2aa14e
(cherry picked from commit 1c16ea4601)
* stable-4.9:
Retry stale file handles on .git/config file
Change-Id: I6db7256dbd1c71b23e1231809642ca21e996e066
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
On a local non-NFS filesystem the .git/config file will be orphaned if
it is replaced by a new process while the current process is reading the
old file. The current process successfully continues to read the
orphaned file until it closes the file handle.
Since NFS servers do not keep track of open files, instead of orphaning
the old .git/config file, such a replacement on an NFS filesystem will
instead cause the old file to be garbage collected (deleted). A stale
file handle exception will be raised on NFS clients if the file is
garbage collected (deleted) on the server while it is being read. Since
we no longer have access to the old file in these cases, the previous
code would just fail. However, in these cases, reopening the file and
rereading it will succeed (since it will open the new replacement file).
Since retrying the read is a viable strategy to deal with stale file
handles on the .git/config file, implement such a strategy.
Since it is possible that the .git/config file could be replaced again
while rereading it, loop on stale file handle exceptions, up to 5 extra
times, trying to read the .git/config file again, until we either read
the new file, or find that the file no longer exists. The limit of 5 is
arbitrary, and provides a safe upper bounds to prevent infinite loops
consuming resources in a potential unforeseen persistent error
condition.
Change-Id: I6901157b9dfdbd3013360ebe3eb40af147a8c626
Signed-off-by: Nasser Grainawi <nasser@codeaurora.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
These methods were introduced for 4.11.1 so we have to silence the API
error adding API in a service release raises.
Change-Id: Ic847cebbed439912d3979ec2ec1809f77a28f61e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The implementation of ObjectIdSerializer, added in change I7599cf8bd,
is not equivalent to the original implementation in Gerrit [1].
The Gerrit implementation provides separate methods to (de)serialize
instances of ObjectId that are known to be non-null. In these methods,
no "marker" is written to the stream. Replacing Gerrit's implementation
with ObjectIdSerializer [2] broke persistent caches because it started
writing markers where they were not expected [3].
Since ObjectIdSerializer is included in JGit 4.11 we can't change the
existing #write and #read methods. Keep those as-is, but extend the
Javadoc to clarify that they support possibly null ObjectId instances.
Add new methods #writeWithoutMarker and #readWithoutMarker to support
the cases where the ObjectId is known to be non-null and the marker
should not be written to the serialization stream.
Also:
- Replace the hard-coded `0` and `1` markers with constants that can
be linked from the Javadocs.
- Include the marker value in the "Invalid flag before ObjectId"
exception message.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/9792
[2] https://gerrit-review.googlesource.com/c/gerrit/+/165851
[3] https://gerrit-review.googlesource.com/c/gerrit/+/165952
Change-Id: Iaf84c3ec32ecf83efffb306fdb4940cc85740f3f
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The WorkingTreeSource produced an ObjectLoader that returned
inconsistent sizes: the file size in getSize(), but then a
correctly filtered smaller stream in openStream(). This resulted
either in an IOE "short read of block" or in an EOFException
depending on the resulting filtered size.
Fix this by ensuring that getSize() does return the size of the
filtered stream.
Bug: 530106
Change-Id: I7c7c85036047dc10030ed29c1d5a6c7f34f2bdff
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-4.10:
Don't subclass ThreadLocal to avoid memory leak in NLS
Set context classloader to null in WorkQueue
Change-Id: Idacf9a15a27f8e1d73357a80ed11a02237eea49e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Make the install command accessible without requiring reflection.
Expose the isEnabled(Repository) API to be able to check if calling the
install command is required for a repository.
Change-Id: I17e6eaefb6afda17fea8162cbf0cb86a20506753
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* changes:
DiffCommand: Open DiffFormatter in try-with-resource
DiffAlgorithms: Open Repository in try-with-resource
DescribeCommandTest: Open FileWriter in try-with-resource
CommitCommand: Open InputStream in try-with-resource
DefaultNoteMerger: Open UnionInputStream in try-with-resource
DirCacheCheckout has a warning about non-localised string "lfs". Other
classes use org.eclipse.jgit.lfs.lib.Constants but that is not visible
to DirCacheCheckout.
Add a new constant in ConfigConstants and use that in DirCacheCheckout.
Replace existing uses of org.eclipse.jgit.lfs.lib.Constants.LFS with
the new constant, except where it is referring to the folder name.
Change-Id: I0f21b951babff9a2e579d68c4de0c62ee4bc23d4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The private method commitNoteMap is in both classes with the same
implementation.
Make it static in AddNoteCommand and reuse it from RemoveNoteCommand.
Change-Id: Ia037bf9efdd94ee7b8d33b41321e9cfd6ea4a6a5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Enable LFS support for the CLI by registering the according filters.
Errors during filter creation must be propagated up the call stack, as a
failure to create a filter should be treated as fatal if the filter is
required.
Change-Id: I3833757839bdda97cd01b6c21c1613d199e2692d
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
As it is right now some streams leak out of the filter construct. This
change clarifies responsibilities and fixes stream leaks
Change-Id: Ib9717d43a701a06a502434d64214d13a392de5ab
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Respect merge=lfs and diff=lfs attributes where required to replace (in
memory) the content of LFS pointers with the actual blob content from
the LFS storage (and vice versa when staging/merging).
Does not implement general support for merge/diff attributes for any
other use case apart from LFS.
Change-Id: Ibad8875de1e0bee8fe3a1dffb1add93111534cae
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Commit 9530c10192 (2018-02-11)
"Add a minimum negotiation feature for fetch" made fetch
negotiation cheaper for repos with huge numbers of
references (we are seeing a 15x reduction in maximum fetch
times for chromium/chromium/src on trans-Pacific links).
But it inadvertently broke the handling of stateless RPC
connections, so fix that here.
Change-Id: I0090aa76ffecc55801ebb833ac2e0c933a4a7c54
Signed-off-by: Terry Parker <tparker@google.com>
If JGit built in LFS support is enabled for the current repository (or
user/system), any existing pre-push hook will cause an exception for the
time beeing, as only a single pre-push hook is supported.
Thus either native pre-push hooks OR JGit built-in LFS support may be
enabled currently, but not both.
Change-Id: Ie7d2b90e26e948d9cca3d05a7a19489488c75895
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This is based on the ObjectIdSerialization class written by Shawn Pearce
for the Gerrit Code Review project in 2009 [1]. As mentioned in the
commit message there, it should be part of core JGit.
This implementation is slightly different to Shawn's version. Rather
than having separate methods for null/non-null ids, single methods are
implemented with @Nullable annotations.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/9792
Change-Id: I7599cf8bd1ecd546e2252783d6d672eb76804060
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Processing of negated rules, like !bin/ was not working correctly: they
were interpreted too broad, resulting in unexpected untracked files
which should actually be ignored
Bug: 409664
Change-Id: I0a422fd6607941461bf2175c9105a0311612efa0
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
The previous wording was ambiguous as to whether these were blocks
requested from the cache (hits + misses) or read from underlying storage
(misses only).
They are in fact recording only misses:
Accumulator#{readBlock,readBlockBytes,readBlockMicros} are only
incremented from BlockBasedFile#readOneBlock, which is only called from
the cache miss path in DfsBlockCache#getOrLoad (line 391).
Change-Id: I0135cd1e76d09c1e28e0f1833b34c312511c66ce
**/ should match only directories, but not files
Change-Id: I885c83e5912cac5bff338ba657faf6bb9ec94064
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
* changes:
RepoCommand: generate relative submodule URLs from absolute URLs.
RepoCommand: don't record new commit if tree did not change
RepoCommand: persist unreadable submodules in .gitmodules
If an object can be found in a non-garbage pack, favor that pack over
paging in the garbage pack's idx and pack content.
Only fall back to garbage packs if an object cannot be found and there
are garbage packs present in the repository. This fallback is
required to correct race conditions during GC.
Change-Id: Ia7c123975bc069b8e6e713eda2d357303b71e329
If a manifest file specifies an absolute URL on the same host on which
the superproject resides, rewrite the URLs to be relative.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: Id616611e5195998fb665c8e7806539a3a02e219a
In cases where a manifest file mixes different remotes, a Gerrit
server process may not have access to all remotes, and won't be able
to produce a full submodule tree.
Preserving this information in .gitmodules will let downstream clients
reconstruct the full tree.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I52f5d3f288e771dca0af2b4dd3f3fa0f940dcf15
Transfer data in chunks of 8k Transferring data byte per byte is slow,
running checkout with CleanFilter on a 2.9MB file takes 20 seconds.
Using a buffer of 8k shrinks this time to 70ms.
Also register the filter commands in a way that the native GIT LFS can
be used alongside with JGit.
Implements auto-discovery of LFS server URL when cloning from a Gerrit
LFS server.
Change-Id: I452a5aa177dcb346d92af08b27c2e35200f246fd
Also-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
When the command is run on a non-bare repository, an instance of
Git is created to execute the commit, and is left open when the
command has finished.
Refactor to not use a class scope Git instance, and make sure it
gets closed before returning.
Change-Id: Ic623ae0fd8b9e264b5dfd434da0de6bb4f910984
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.10:
Fix ssh host name handling for Jsch
Jsch overrides the port in the URI with the one in ~/.ssh/config
Change-Id: I860fc61ceb12ae792b1ee7421046ecd32373b9f8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
These problem usually occur when you subclass ThreadLocal (usually to
implement initialValue). Those classes reference the webapp's
classloader. The ThreadLocal subclass in turn is referenced by each
Thread instance (that's how ThreadLocals are implemented, they have a
"helper-Map" in each Thread instance, so the leak is actually not a tiny
Random instance but the whole webapp's classloader with a bunch of class
definitions and statically referenced parts of the webapp.
Bug: 449321
Change-Id: Ie7a8b0b90e40229e2471202f2a12637b9e0b1d11
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.9:
Fix ssh host name handling for Jsch
Jsch overrides the port in the URI with the one in ~/.ssh/config
Change-Id: Iff9076f65e767bbe8df016337b631bdaeb40ad98
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
If we give Jsch access to the ssh config file, we must _not_ resolve
the host name from the alias. Instead we must give the alias (i.e.,
the host name as is in the URI) to Jsch, so that it finds the same
ssh config entry.
Otherwise if the hostname in the URI, which is taken as an alias in
ssh config ("Host" line), is unequal to the "Hostname" line, and
there happens to be another ssh config entry with that translated
host name as alias, Jsch will pick up that second entry, and we end
up with a strange mixture of both.
Add tests for this case.
Bug: 531118
Change-Id: I249d8c073b0190ed110a69dca5b9be2a749822c3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Jsch unconditionally overwrites the port from the ssh config
file (if a port is specified there), even if the URI explicitly does
give a different port.
Fix this, and add tests.
Change-Id: I7b014543c7ece26270e366db39d7647f82d64f0d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The InitCommand returns a Git that is instantiated with the newly
created Repository, but the Repository is not closed with the Git
resulting in resource leaks.
Create the Git with `closeRepo` set to true, such that the Repository
is also closed when the Git is closed.
Adjust the tests to use try-with-resource on the Git instance.
Change-Id: Ib26e7428c7d8840956d1edb09e53b93e23e6fe5a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Android an Chrome have several repos with >300k refs. We sometimes see
negotiations of >100k rounds. This change provides a "minimal negotiation"
feature on the client side that limits how many "have" lines the client
sends. The client extracts the current SHA-1 values for the refs in its
wants set, and terminates negotiation early when all of those values have
been sent as haves. If a new branch is being fetched then that set will
be empty and the client will terminate after current default minimum
of two rounds.
This feature is gated behind a "fetch.useminimalnegotiation" configuration
flag, which defaults to false.
Change-Id: Ib12b095cac76a59da6e8f72773c4129e3b32ff2b
Signed-off-by: Terry Parker <tparker@google.com>
Add fetch statistics for the counts of advertised refs, wants and haves.
Also add the duration in milliseconds for the negotiation phase. For
non-bidirectional transports like HTTP, this is the time for the final
round that sends the pack back to the user.
Change-Id: I1af7ffd3cb7b62182340682e2a243691ea24ec2e
Signed-off-by: Terry Parker <tparker@google.com>
This would allow compact and GC process to clean up duplicate ref names in the reftables.
Change-Id: I2b9df0bf72dba63cc3525e374982e60559a776c2
Signed-off-by: Minh Thai <mthai@google.com>
When CleanCommand is collecting the files and folders to be deleted
it may happen that the list of directories contains obsolete entries.
E.g. a folder and its parent folder may be in the list. Only the
parent folder would be sufficient.
This was a reason for hitting FileNotFoundExceptions when finally
trying to delete the files and folders. Improve CleanCommand
to ignore files to be deleted which are already gone.
Bug: 514434
Change-Id: I10caa01bfb9cec5967dfdaea50c6e4a713eeeabd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
After packaging references, the folders containing these references are
not deleted. In a busy repository, this causes operations to slow down
as traversing the references tree becomes longer.
Delete empty reference folders after the loose references have been
packed.
To avoid deleting a folder that was just created by another concurrent
operation, only delete folders that were not modified in the last 30
seconds.
Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: Ie79447d6121271cf5e25171be377ea396c7028e0
Bazel@HEAD supports Java 9.
The current code has one single issue with Java 9 compliance: the usage
of javax.xml.bind.DatatypeConverter class for printHexBinary() method.
This class is not available on Java 9. One alternative is to use guava
library. Something similar was done here: [1]. But unlike the case with
checkstyle library, JGit currently doesn't use guava. Instead, we add
java.xml.bind module with --add-modules compiler option.
To build (or test) with Java 9, build custom bazel version and issue:
$ bazel --host_javabase=/usr/lib64/jvm/java-9-openjdk build \
--javacopt='--release 9' \
--java_toolchain=@bazel_tools//tools/jdk:toolchain_jdk9 :all
The Java 9 support is backwards compatible.
* [1] https://github.com/checkstyle/checkstyle/issues/5027
Change-Id: I2c5203fc4e65885ce7b210f824fda85ba6d6c51d
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
The Files.exists method has noticeably poor performance in JDK 8 and can
slow an application significantly when used to check files that do not
actually exist. The same goes for Files.notExists, Files.isDirectory and
Files.isRegularFile [1].
Replace them with their File counterpart.
[1] https://rules.sonarsource.com/java/tag/performance/RSPEC-3725
Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: I89d23b9cc74bec8e05f6b7f3e49bfd967dbb6373
Avoid converting path to file to then reconvert it to path.
Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: I6a8c3ca9b83bf9b0eead9506938f5d68b27a76f5
continue is unnecessary when it is the last statement in a loop
Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: I12af9f9a0bb2fd7fc0239f1f3b59fb8e64e1f351
This doesn't handle the really hard thing, which is merging spurious
conflicts inside .gitmodules files. That's OK: git.git doesn't
either. Users can resolve the conflict themselves and then commit
the merge.
Previously, jgit would crash when attempting to merge conflicting
submodule changes. Even if there was no conflict, after a merge which
adds submodules, the repository would have been missing empty
directories for newly-added submodules.
This patch fixes the crash, and adds the empty directories where
necessary. It ensures that the index is in a conflicted state when
submodule changes conflict.
Reported-by: Alexey Korobkov
Bug: 494551
Change-Id: I79db6798c2bdcc1159b5b2589b02da198dc906a1
Signed-off-by: David Turner <dturner@twosigma.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Luca Milanesio