* stable-4.9:
Fix ObjectUploadListener#close
Fix error handling in FileLfsServlet
ObjectDownloadListener#onWritePossible: Make code spec compatible
ObjectDownloadListener: Return from onWritePossible when data is written
Fix IOException when LockToken#close fails
Change-Id: Ib7d01cb0ece8b259156855045a53b8baf3fa2968
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.8:
Fix ObjectUploadListener#close
Fix error handling in FileLfsServlet
ObjectDownloadListener#onWritePossible: Make code spec compatible
ObjectDownloadListener: Return from onWritePossible when data is written
Fix IOException when LockToken#close fails
Change-Id: Id8eb635094336567d9f3c28ec985cd5127d31632
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.7:
Fix ObjectUploadListener#close
Fix error handling in FileLfsServlet
ObjectDownloadListener#onWritePossible: Make code spec compatible
ObjectDownloadListener: Return from onWritePossible when data is written
Fix IOException when LockToken#close fails
Change-Id: Iad9836811be034cf992ea25dad4409addba75115
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This happened if the LockTokens hard link was already deleted earlier.
Bug: 531759
Change-Id: Idc84bd695fac1a763b3cbb797c9c4c636a16e329
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.9:
Fix NoSuchFileException during directory cleanup in RefDirectory
Externalize warning message in RefDirectory.delete()
Suppress warning for trying to delete non-empty directory
Change-Id: Ied2d7a70c5544ff012687a1aeb6e07394d9bd6fd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.8:
Fix NoSuchFileException during directory cleanup in RefDirectory
Externalize warning message in RefDirectory.delete()
Suppress warning for trying to delete non-empty directory
Change-Id: I5e6cc35f3673545e7ff857e6ed0bcd2c44e50316
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.7:
Fix NoSuchFileException during directory cleanup in RefDirectory
Externalize warning message in RefDirectory.delete()
Suppress warning for trying to delete non-empty directory
Change-Id: I9ec6352b5ff57aa1a3380079dc9165890cc76d49
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Change-Id: Icec16c01853a3f5ea016d454b3d48624498efcce
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit 5e68fe245f)
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This is actually a fairly common occurrence; deleting the parent
directories can work only if the file deleted was the last one
in the directory.
Bug: 537872
Change-Id: I86d1d45e1e2631332025ff24af8dfd46c9725711
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
(cherry picked from commit d9e767b431)
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
FS_POSIX.createNewFile(File) failed to properly implement atomic file
creation on NFS using the algorithm [1]:
- name of the hard link must be unique to prevent that two processes
using different NFS clients try to create the same link. This would
render nlink useless to detect if there was a race.
- the hard link must be retained for the lifetime of the file since we
don't know when the state of the involved NFS clients will be
synchronized. This depends on NFS configuration options.
To fix these issues we need to change the signature of createNewFile
which would break API. Hence deprecate the old method
FS.createNewFile(File) and add a new method createNewFileAtomic(File).
The new method returns a LockToken which needs to be retained by the
caller (LockFile) until all involved NFS clients synchronized their
state. Since we don't know when the NFS caches are synchronized we need
to retain the token until the corresponding file is no longer needed.
The LockToken must be closed after the LockFile using it has been
committed or unlocked. On Posix, if core.supportsAtomicCreateNewFile =
false this will delete the hard link which guarded the atomic creation
of the file. When acquiring the lock fails ensure that the hard link is
removed.
[1] https://www.time-travellers.org/shane/papers/NFS_considered_harmful.html
also see file creation flag O_EXCL in
http://man7.org/linux/man-pages/man2/open.2.html
Change-Id: I84fcb16143a5f877e9b08c6ee0ff8fa4ea68a90d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When core.supportsAtomicCreateNewFile was set to false and the
repository was located on a filesystem which doesn't support the file
attribute "unix:nlink" then FS_POSIX#createNewFile may report an error
even if everything was ok. Modify FS_POSIX#createNewFile to silently
ignore this situation. An example of such a filesystem is sshfs where
reading "unix:nlink" always returns 1 (instead of throwing a exception).
Bug: 537969
Change-Id: I6deda7672fa7945efa8706ea1cd652272604ff19
Also-by: Thomas Wolf <thomas.wolf@paranor.ch>
I88304d34c and Ia555bce00 modified the way errors are handled when
trying to delete non-empty reference folders. Before, this error was
silently ignored as it was considered an expected output. Now, every
failed folder delete is logged which can be noisy.
Ignore the DirectoryNotEmptyException but log any other error avoiding
deletion of an eligible folder.
Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: I194512f67885231d62c03976ae683e5cc450ec7c
* stable-4.9:
Ignore API warnings
Fix photon target platform to use photon version of org.eclipse.osgi
Update Photon orbit repository to R20180606145124
Suppress warning for trying to delete non-empty directory
Fix fetching with duplicate ref updates
Fetch(Process): should tolerate duplicate refspecs
FetchCommandTest: test add/update/delete fetch
Change-Id: I2a83c059b7014c2a9e6267c963422c7785b23f17
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The following commits introduced in stable-4.5 and stable-4.9
introduced some minor API additions in service releases.
f7ceeaa2 FileRepository: Add pack-based inserter implementation
085d1f95 Make PackInserter public
10e65cb4 Fix LockFile semantics when running on NFS
Change-Id: I4afed7e0395cf93d828e671080e3ec9ddf20987d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This is actually a fairly common occurrence; deleting the parent
directories can work only if the file deleted was the last one
in the directory.
Bug: 537872
Change-Id: I86d1d45e1e2631332025ff24af8dfd46c9725711
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
If packed refs are used, duplicate updates result in an exception
because JGit tries to lock the same lock file twice. With non-atomic
ref updates, this used to work, since the same ref would simply be
locked and updated twice in succession.
Let's be more lenient in this case and remove duplicates before
trying to do the ref updates. Silently skip duplicate updates
for the same ref, if they both would update the ref to the same
object ID. (If they don't, behavior is undefined anyway, and we
still throw an exception.)
Add a test that results in a duplicate ref update for a tag.
Bug: 529400
Change-Id: Ide97f20b219646ac24c22e28de0c194a29cb62a5
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Bug: 529314
Change-Id: I91eaeda8a988d4786908fba6de00478cfc47a2a2
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
* stable-4.9:
Fix GC run in foreground to not use executor
Change-Id: Ib4d76125fca7eec9e88666688b5e614e7e20dde7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.8:
Fix GC run in foreground to not use executor
Change-Id: Id9d864a8e727fefa35ca87eccb4e3801eb689c3c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.7:
Fix GC run in foreground to not use executor
Change-Id: Ib150d132e2ce055d36ddffb2dbc37b5cb355e77a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Since I3870cadb4, GC task was always delegated to an executor even when
background option was set to false. This was an issue because if more
than one GC object was instantiated and executed in parallel, only one GC
was actually running because of the single thread executor.
Change-Id: I8c587d22d63c1601b7d75914692644a385cd86d6
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
* stable-4.9:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: Ie7029bc91621af32e7bfd2e0d76a424b991b1995
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.8:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: I6ab2b951dd94a9fc1c4f5283847a3e2ec37d0895
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* stable-4.7:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: Iadc8275fbaa3d6f7d08a96ab66d49f392f6aab78
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Remove completely the empty directories under refs/<namespace>
including the first level partition of the changes, when they are
completely empty.
Bug: 536777
Change-Id: I88304d34cc42435919c2d1480258684d993dfdca
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Get the full IOException of the reason why a directory
cannot be removed during GC.
Change-Id: Ia555bce009fa48087a73d677f1ce3b9c0b685b57
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
After packaging references, the folders containing these references are
not deleted. In a busy repository, this causes operations to slow down
as traversing the references tree becomes longer.
Delete empty reference folders after the loose references have been
packed.
To avoid deleting a folder that was just created by another concurrent
operation, only delete folders that were not modified in the last 30
seconds.
Signed-off-by: Hector Oswaldo Caballero <hector.caballero@ericsson.com>
Change-Id: Ie79447d6121271cf5e25171be377ea396c7028e0
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Log as warning when an attempt to remove a directory
fails. This helps troubleshooting some bugs like the GC leaving
behind empty directories.
Change-Id: Idb94ce17f8be9668a970c7ecae31436bf434073c
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
* stable-4.9:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: Ibd8f2a041b0de6e008a1ea84b92823f8cbc6e3d2
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.8:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: Id6a85804695d5dcb32f26ed1d861b7c93577c5e4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.7:
ResolveMerger: Fix encoding with string; use bytes
Change-Id: If17328fbd101d596a8a16d9c4a190e9b6e120902
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This change fixes the issue [1]. Before this fix, a merge involving
the caching of consecutive yet similar filenames with Norwegian
characters [2] used to throw an IllegalStateException: Duplicate
stages not allowed. This was caused by inaccurate decoding of the
filenames, using string values assuming default encoding. In the
toString method of DirCacheEntry, used before through getPathString,
UTF-8 encoding is used, but the end result becomes default encoding,
through Object's default toString usage. The special characters in
those two consecutive (particular) filenames [2] were becoming the
very same decoded /single character, lending consecutive -but then
identical- filenames. Thus the perceived duplicate 0-staging of the
file(s).
Replace getPathString usage with getRawPath for this specific case,
or use byte array representations of cached entries instead of string.
Adding a test for this change is not possible, as there is no known
way to change the default encoding for filenames such as [2] (e.g.).
JGitTestUtil does write file contents through UTF-8, but encoding like
so does not apply to the actual file name. Hence there is no way to
create files with names properly made of special characters such as
[2]'s. And the test that is necessary for this case assumes such
Norwegian (or similar characters) filenames. Changing the default
locale programmatically in a test has no effect either. And changing
the LANG value passed to the JVM is only possible upon starting it.
[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=9153
[2] <=>
(...)
"a/b/SíÒr-Norge.map",
"a/b/Sør-Norge.map",
(...)
Change-Id: Ib9f2f5297932337c9817064cc09d9f774dd168f4
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
From the javadoc for Files.list:
"The returned stream encapsulates a DirectoryStream. If timely disposal
of file system resources is required, the try-with-resources construct
should be used to ensure that the stream's close method is invoked
after the stream operations are completed."
This is the only call to Files#newDirectoryStream that is not already in
a try-with-resources.
Change-Id: I91e6c56b5d74e8435457ad6ed9e6b4b24d2aa14e
(cherry picked from commit 1c16ea4601)
David Pursehouse