Commit Graph

4013 Commits

Author SHA1 Message Date
Luca Milanesio 5a8ad44208 Do not ignore path deletion errors
Log as warning when an attempt to remove a directory
fails. This helps troubleshooting some bugs like the GC leaving
behind empty directories.

Change-Id: Idb94ce17f8be9668a970c7ecae31436bf434073c
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
2018-07-08 23:44:17 +01:00
David Pursehouse b6a6ba10c7 Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
  ResolveMerger: Fix encoding with string; use bytes

Change-Id: Id6a85804695d5dcb32f26ed1d861b7c93577c5e4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-07-04 14:09:53 +09:00
David Pursehouse c296342333 Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
  ResolveMerger: Fix encoding with string; use bytes

Change-Id: If17328fbd101d596a8a16d9c4a190e9b6e120902
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2018-07-04 14:09:01 +09:00
Marco Miller 1dcb0688c7 ResolveMerger: Fix encoding with string; use bytes
This change fixes the issue [1]. Before this fix, a merge involving
the caching of consecutive yet similar filenames with Norwegian
characters [2] used to throw an IllegalStateException: Duplicate
stages not allowed. This was caused by inaccurate decoding of the
filenames, using string values assuming default encoding. In the
toString method of DirCacheEntry, used before through getPathString,
UTF-8 encoding is used, but the end result becomes default encoding,
through Object's default toString usage. The special characters in
those two consecutive (particular) filenames [2] were becoming the
very same decoded /single character, lending consecutive -but then
identical- filenames. Thus the perceived duplicate 0-staging of the
file(s).

Replace getPathString usage with getRawPath for this specific case,
or use byte array representations of cached entries instead of string.

Adding a test for this change is not possible, as there is no known
way to change the default encoding for filenames such as [2] (e.g.).
JGitTestUtil does write file contents through UTF-8, but encoding like
so does not apply to the actual file name. Hence there is no way to
create files with names properly made of special characters such as
[2]'s. And the test that is necessary for this case assumes such
Norwegian (or similar characters) filenames. Changing the default
locale programmatically in a test has no effect either. And changing
the LANG value passed to the JVM is only possible upon starting it.

[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=9153

[2] <=>
(...)
"a/b/SíÒr-Norge.map",
"a/b/Sør-Norge.map",
(...)

Change-Id: Ib9f2f5297932337c9817064cc09d9f774dd168f4
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
2018-06-21 15:38:56 -04:00
Dave Borowitz 5fe8e31d43 Ensure DirectoryStream is closed promptly
From the javadoc for Files.list:

"The returned stream encapsulates a DirectoryStream. If timely disposal
of file system resources is required, the try-with-resources construct
should be used to ensure that the stream's close method is invoked
after the stream operations are completed."

This is the only call to Files#newDirectoryStream that is not already in
a try-with-resources.

Change-Id: I91e6c56b5d74e8435457ad6ed9e6b4b24d2aa14e
(cherry picked from commit 1c16ea4601)
2018-06-09 17:34:29 +09:00
Matthias Sohn dd9a14a762 Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
  Retry stale file handles on .git/config file

Change-Id: Ib029b5536c038190626e7a7ff43b70f0a5673721
2018-05-10 12:30:05 +02:00
Matthias Sohn d5ff94d575 Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
  Retry stale file handles on .git/config file

Change-Id: Ib665c094b28eefb8236752bb273de0c2d9bf9578
2018-05-10 12:08:46 +02:00
Matthias Sohn 1887c83477 Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
  Retry stale file handles on .git/config file

Change-Id: If5a21d38224528edfc551b3216daca6a2582e3ac
2018-05-10 11:59:56 +02:00
Matthias Sohn 9253a6c5ca Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
  Retry stale file handles on .git/config file

Change-Id: Ib6e6ec0846c3ef261ec1016bfa6d26d2eadc3f26
2018-05-10 11:39:52 +02:00
Nasser Grainawi d13918310f Retry stale file handles on .git/config file
On a local non-NFS filesystem the .git/config file will be orphaned if
it is replaced by a new process while the current process is reading the
old file. The current process successfully continues to read the
orphaned file until it closes the file handle.

Since NFS servers do not keep track of open files, instead of orphaning
the old .git/config file, such a replacement on an NFS filesystem will
instead cause the old file to be garbage collected (deleted).  A stale
file handle exception will be raised on NFS clients if the file is
garbage collected (deleted) on the server while it is being read.  Since
we no longer have access to the old file in these cases, the previous
code would just fail. However, in these cases, reopening the file and
rereading it will succeed (since it will open the new replacement file).
Since retrying the read is a viable strategy to deal with stale file
handles on the .git/config file, implement such a strategy.

Since it is possible that the .git/config file could be replaced again
while rereading it, loop on stale file handle exceptions, up to 5 extra
times, trying to read the .git/config file again, until we either read
the new file, or find that the file no longer exists. The limit of 5 is
arbitrary, and provides a safe upper bounds to prevent infinite loops
consuming resources in a potential unforeseen persistent error
condition.

Change-Id: I6901157b9dfdbd3013360ebe3eb40af147a8c626
Signed-off-by: Nasser Grainawi <nasser@codeaurora.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2018-05-10 11:13:32 +02:00
Thomas Wolf 185e53bce4 Fix ssh host name handling for Jsch
If we give Jsch access to the ssh config file, we must _not_ resolve
the host name from the alias. Instead we must give the alias (i.e.,
the host name as is in the URI) to Jsch, so that it finds the same
ssh config entry.

Otherwise if the hostname in the URI, which is taken as an alias in
ssh config ("Host" line), is unequal to the "Hostname" line, and
there happens to be another ssh config entry with that translated
host name as alias, Jsch will pick up that second entry, and we end
up with a strange mixture of both.

Add tests for this case.

Bug: 531118
Change-Id: I249d8c073b0190ed110a69dca5b9be2a749822c3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-02-14 13:39:28 +01:00
Thomas Wolf 891103609d Jsch overrides the port in the URI with the one in ~/.ssh/config
Jsch unconditionally overwrites the port from the ssh config
file (if a port is specified there), even if the URI explicitly does
give a different port.

Fix this, and add tests.

Change-Id: I7b014543c7ece26270e366db39d7647f82d64f0d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-02-14 13:32:37 +01:00
Thomas Wolf 3d6455331d Minor fixes in three error messages
* Fix "can not" -> "cannot" in two messages
* Re-word "Cannot mkdir" to "Cannot create directory"

Change-Id: Ide0cec55eeeebd23bccc136257c80f47638ba858
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-01-31 14:29:49 +09:00
Thomas Wolf bea3b46678 Revert handling of ssh IdentityFile to pre-4.9 behavior
Jsch caches keys (aka identities) specified in ~/.ssh/config via
IndentityFile only for the current Jsch Session. This results in
multiple password prompts for successive sessions.

Do the handling of IdentityFile exclusively in JGit, as it was before
4.9. JGit uses different Jsch instances per host and caches the
IdentityFile there, allowing it to be re-used in different sessions
for the same host.

* Add comments to explain this.
* Move the JschBugFixingConfig from OpenSshConfig to
  JschConfigSessionFactory to have all these Jsch work-arounds
  in one place.
* Make that config hide the IdentityFile config from Jsch to avoid
  that Jsch overrides the JGit behavior.

Bug: 529173
Change-Id: Ib36c34a2921ba736adeb64de71323c2b91151613
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2018-01-12 12:56:00 +01:00
Dmitry Pavlenko 279eaf0690 Strings#convertGlob: fix escaping of patterns like [\[].
Originally the patterns were escaped twice leading
to wrong matching results.

Bug: 528886
Change-Id: I26e201b4b0ef51cac08f940b76f381260fa925ca
Signed-off-by: Dmitry Pavlenko <pavlenko@tmatesoft.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-21 18:33:22 -05:00
Dave Borowitz 085d1f9599 Make PackInserter public
The intent with the setCompressionLevel and checkExisting methods (which
are already public) is for callers to be able to call them, but they
can't do that if the class itself is not public.

Change-Id: I014044fec3bfa1d33775500345efd60eb5d45bde
2017-12-21 09:46:55 -05:00
Dave Borowitz 43ef5dabf1 PackInserter: Ensure objects are written at the end of the pack
When interleaving reads and writes from an unflushed pack, we forgot to
reset the file pointer back to the end of the file before writing more
new objects. This had at least two unfortunate effects:
  * The pack data was potentially corrupt, since we could overwrite
    previous portions of the file willy-nilly.
  * The CountingOutputStream would report more bytes read than the size
    of the file, which stored the wrong PackedObjectInfo, which would
    cause EOFs during reading.

We already had a test in PackInserterTest which was supposed to catch
bugs like this, by interleaving reads and writes. Unfortunately, it
didn't catch the bug, since as an implementation detail we always read a
full buffer's worth of data from the file when inflating during
readback. If the size of the file was less than the offset of the object
we were reading back plus one buffer (8192 bytes), we would completely
accidentally end up back in the right place in the file.

So, add another test for this case where we read back a small object
positioned before a large object. Before the fix, this test exhibited
exactly the "Unexpected EOF" error reported at crbug.com/gerrit/7668.

Change-Id: I74f08f3d5d9046781d59e5bd7c84916ff8225c3b
2017-12-20 12:43:31 -05:00
Dave Borowitz f63ee965d4 ObjectInserter: Add warning about mixing read-back with writes
Change-Id: Ib0460d3c7df315d86f9adca5f66a8fd4c39e4060
2017-12-20 12:16:40 -05:00
Matthias Sohn 1e56842742 Prepare 4.9.3-SNAPSHOT builds
Change-Id: Ife3f2b0b5407227f89ded42358adbf01d53e14cf
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-16 03:49:03 +01:00
Matthias Sohn 24b7e91264 JGit v4.9.2.201712150930-r
Change-Id: I013964045d532659a4be3b81d6612b59bc9ffb14
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-15 15:29:36 +01:00
Thomas Wolf c86327979f Write packed-refs directly when cloning
When we are cloning we have no refs at all yet, and there cannot
(or at least should not) be any other thread doing something with
refs yet.

Locking loose refs is thus not needed, since there are no loose
refs yet and nothing should be trying to create them concurrently.

Let's skip the whole loose ref locking when we are cloning a repository.
As a result, JGit will write the refs directly to the packed-refs
file, and will not create the refs/remotes/ directories nor the
lock files underneath when cloning and packed refs are used. Since
no lock files are created, any problems on case-insensitive file
systems with tag or branch names that differ only in case are avoided
during cloning.

Detect if we are cloning based on the following heuristics:
* HEAD is a dangling symref
* There is no loose ref
* There is no packed-refs file

Note, however, that there may still be problems with such tag or
branch names later on. This is primarily a five-minutes-past-twelve
stop-gap measure to resolve the referenced bug, which affects the
Oxygen.2 release.

Bug: 528497
Change-Id: I57860c29c210568165276a123b855e462b6a107a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-15 14:14:03 +01:00
David Pursehouse ec7f88eec8 Config: Remove the include functionality
The Config class must be safe to run against untrusted input files.
Reading arbitrary local system paths using include.path is risky for
servers, including Gerrit Code Review.

This was fixed on master [1] by making "readIncludedConfig" a noop
by default. This allows only FileBasedConfig, which originated from
local disk, to read local system paths.

However, the "readIncludedConfig" method was only introduced in [2]
which was needed by [3], both of which are only on the master branch.
On the stable branch only Config supports includes. Therefore this
commit simply disables the include functionality.

[1] https://git.eclipse.org/r/#/c/113371/
[2] https://git.eclipse.org/r/#/c/111847/
[3] https://git.eclipse.org/r/#/c/111848/

Bug: 528781
Change-Id: I9a3be3f1d07c4b6772bff535a2556e699a61381c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-15 08:58:00 +09:00
David Pursehouse c40e150790 InMemoryRepository: Make inner class MemObjDatabase static
FindBugs reports:

  This class is an inner class, but does not use its embedded reference
  to the object which created it. This reference makes the instances
  of the class larger, and may keep the reference to the creator object
  alive longer than necessary. If possible, the class should be made
  static.

Change-Id: I9f49de32b4cd81b7ef1239b390353689263bf66e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-12 00:39:21 -05:00
David Turner 623851ec16 TransportCommand#setTimeout: Specify units for timeout in Javadoc
Change-Id: Iadf0e2c6483794e43a5326f6be6bb34c19b00564
Signed-off-by: David Turner <dturner@twosigma.com>
2017-12-12 13:25:05 +09:00
Matthias Sohn 9d79dc95e2 Fix typo in key of a JGitText externalized string
Change-Id: I0d22e24a0aa3b17339ef68849554f7c99b350dde
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-11 09:35:15 +09:00
Dmitry Pavlenko 7ccf38adc6 Fix IllegalThreadStateException if stderr closed without exiting
If some process executed by FS#readPipe lived for a while after
closing stderr, FS#GobblerThread#run failed with an
IllegalThreadStateException exception when accessing p.exitValue()
for the process which is still alive.

Add Process#waitFor calls to wait for the process completion.

Bug: 528335
Change-Id: I87e0b6f9ad0b995dbce46ddfb877e33eaf3ae5a6
Signed-off-by: Dmitry Pavlenko <pavlenko@tmatesoft.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-10 18:00:55 +01:00
David Pursehouse 2f0a880bd6 DirCacheCheckout#processEntry: Fix typo in javadoc
Change-Id: Ie08a6275f4656ebb46abe8748b73b0fb52a98a35
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-12-06 20:00:21 +09:00
Matthias Sohn 68c77a4d39 Prepare 4.9.2-SNAPSHOT builds
Change-Id: I5879ad4aee94ff6783b5589728912117f2495dd3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-03 14:17:43 +01:00
Matthias Sohn a3588cbb2a JGit v4.9.1.201712030800-r
Change-Id: I8bf477778c9dac41cb65233a9e7d590531a836b7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-03 13:59:36 +01:00
Matthias Sohn d2a23afb53 Silence API warnings for reintroduced ResolveMerger#processEntry
This was silenced before but suppression was unintentionally lost in
merge commit 6858339c1e.
This method was removed in 4.9.0 and reintroduced in 4.9.1 to avoid
breaking EMF compare versions which were built against older versions.

See: abf420302b
Change-Id: I152d58ac885e044bcab682b9423f6cc83b667989
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-12-03 13:58:34 +01:00
Hector Caballero bac4d32d39 GC: Delete stale temporary packs and indexes
When a GC operation is interrupted, temporary packs and indexes can be
left on the pack folder. In big, busy repositories this can lead to
significant amounts of wasted disk space if this interruption is done
with a certain frequency.

Remove stale temporary packs and indexes at the end of the GC process so
they do not accumulate. To avoid interfering with a possible concurrent
JGit GC process in the same repository, only delete temporary files that
are older than one day.

Change-Id: If9b6c1e57fac8a6a0ecc0a703089634caba4caae
Signed-off-by: Hector Caballero <hector.caballero@ericsson.com>
2017-11-24 05:13:24 -05:00
Matthias Sohn 03abd1dff2 Ignore warning for minor version change without API change
- this is a new warning option in Eclipse 4.7 and higher
- we always change version of all bundles in a release to keep release
engineering simple

Change-Id: Ic7523d77b67b2802f1bab3bc70af250d712a034f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-24 01:12:14 +01:00
Matthias Sohn 6858339c1e Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
  Silence boxing warning
  Prepare 4.5.5-SNAPSHOT builds
  JGit v4.5.4.201711221230-r
  Fix LockFile semantics when running on NFS
  Honor trustFolderStats also when reading packed-refs
  Prepare 4.5.4-SNAPSHOT builds
  JGit v4.5.3.201708160445-r

Change-Id: I7cf2e48934195430b3945b6d74b092f93a3ccd36
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-24 01:08:23 +01:00
Matthias Sohn 4ac790fda8 Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
  Silence boxing warning
  Prepare 4.5.5-SNAPSHOT builds
  JGit v4.5.4.201711221230-r
  Fix LockFile semantics when running on NFS
  Honor trustFolderStats also when reading packed-refs
  Prepare 4.5.4-SNAPSHOT builds
  JGit v4.5.3.201708160445-r

Change-Id: Iaa99ec84594baf733c993c2d6768281ff14f545a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 23:30:13 +01:00
Matthias Sohn 5f79ef0ba4 Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
  Silence boxing warning
  Prepare 4.5.5-SNAPSHOT builds
  JGit v4.5.4.201711221230-r
  Fix LockFile semantics when running on NFS
  Honor trustFolderStats also when reading packed-refs
  Prepare 4.5.4-SNAPSHOT builds
  JGit v4.5.3.201708160445-r

Change-Id: I8f6bc09540727c6273d22775a9f9ca382a729c9b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 23:07:01 +01:00
Matthias Sohn 182d3b3dad Silence boxing warning
Change-Id: I36c40eb91ce0c51f89b47911fa14beffcbc0a7cd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 21:34:37 +01:00
Matthias Sohn 3ac44f1b38 Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
  Prepare 4.5.5-SNAPSHOT builds
  JGit v4.5.4.201711221230-r
  Fix LockFile semantics when running on NFS
  Honor trustFolderStats also when reading packed-refs
  Prepare 4.5.4-SNAPSHOT builds
  JGit v4.5.3.201708160445-r

Change-Id: Ie9c8e0d9172c8d53f075c284bf2a9677980d8dfb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 21:34:08 +01:00
Matthias Sohn 7608de5e5d Prepare 4.5.5-SNAPSHOT builds
Change-Id: I71f946f2875716670a2d74c21a8ab38a1f53a25c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 19:27:22 +01:00
Matthias Sohn 6372c60ab2 JGit v4.5.4.201711221230-r
Change-Id: Ia1079da239c5b3fde1ba8d2acc4e465a46297b4d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-22 18:31:24 +01:00
Christian Halstrick 10e65cb4fa Fix LockFile semantics when running on NFS
When running on NFS there was a chance that JGits LockFile
semantic is broken because File#createNewFile() may allow
multiple clients to create the same file in parallel. This
change provides a fix which is only used when the new config
option core.supportsAtomicCreateNewFile is set to false. The
default for this option is true. This option can only be set in the
global or the system config file. The repository config file is not
taken into account in this case.

If the config option core.supportsAtomicCreateNewFile is true
then File#createNewFile() is trusted and the behaviour doesn't
change.

But if core.supportsAtomicCreateNewFile is set to false then after
successful creation of the lock file a hardlink to that lock file is
created and the attribute nlink of the lock file is checked to be 2. If
multiple clients manage to create the same lock file nlink would be
greater than 2 showing the error.

This expensive workaround is described in
 https://www.time-travellers.org/shane/papers/NFS_considered_harmful.html
section III.d) "Exclusive File Creation"

Change-Id: I3d2cc48d8eb280d5f7039eb94da37804f903be6a
2017-11-22 18:15:11 +01:00
Christian Halstrick 218cf3403d Honor trustFolderStats also when reading packed-refs
Then list of packed refs was cached in RefDirectory based on mtime of
the packed-refs file. This may fail on NFS when attributes are cached.
A cached mtime of the packed-refs file could cause JGit to trust the
cached content of this file and to overlook that the file is modified.

Honor the config option trustFolderStats and always read the packed-refs
content if the option is false. By default this option is set to true
and this fix is not active.

Change-Id: I2b65cfaa8f4aba2efbf8a5e865d3f09f927e2eec
2017-11-21 21:21:22 +01:00
Thomas Wolf 5284cc1bf7 Yet another work-around for a Jsch bug: timeouts
Jsch 0.1.54 passes on the values from ~/.ssh/config for
"ServerAliveInterval" and "ConnectTimeout" as read from
the config file to java.net.Socket.setSoTimeout(). That
method expects milliseconds, but the values in the config
file are seconds!

The missing conversion in Jsch means that the timeout is
set way too low, and if the server doesn't respond within
that very short time frame, Jsch kills the connection and
then throws an exception with a message such as "session is
down" or "timeout in waiting for rekeying process".

As a work-around, do the conversion to milliseconds in the
Jsch-facing Config interface of OpenSshConfig. That way Jsch
already gets these values as milliseconds.

Bug: 526867
Change-Id: Ibc9b93f7722fffe10f3e770dfe7fdabfb3b97e74
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-11-20 22:44:23 +01:00
Matthias Sohn 29c5f49f63 Fix NPE in TransportGitSsh.ExtSession.exec()
Bug: 527383
Change-Id: I965860b4e0e9aa1adf6a1bb0735940cbd22adced
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-11-20 00:35:37 -05:00
Thomas Wolf f67af4e16b Work around a Jsch bug: ensure the user name is set from URI
JSch unconditionally overrides the user name given in the connection
URI by the one found in ~/.ssh/config (if that does specify one for
the used host). If the SSH config file has a different user name,
we'll end up using the wrong name, which typically results in an
authentication failure or in Eclipse/EGit asking for a password for
the wrong user.

Unfortunately there is no way to prevent or circumvent this Jsch
behavior up front; it occurs already in the Session constructor at
com.jcraft.jsch.Session() and the Session.applyConfig() method. And
while there is a Session.setUserName() that would enable us to correct
this, that latter method has package visibility only.

So resort to reflection to invoke that setUserName() method to ensure
that Jsch uses the user name from the URI, if there is one.

Bug: 526778
Change-Id: Ia327099b5210a037380b2750a7fd76ff25c41a5a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-11-10 10:16:58 +01:00
Matthias Sohn 031646963f Merge "Reintroduce protected method which removal broke EMF Compare" into stable-4.9 2017-11-02 12:59:11 -04:00
Dave Borowitz 678c99c057 PackInserter: Implement newReader()
Change-Id: Ib9e7f6439332eaed3d936f895a5271a7d514d3e9
2017-11-01 13:00:24 -04:00
Dave Borowitz 080b4770e7 Move some strings from DfsText to JGitText
Change-Id: I60050e5127d12b6139d81859dba929fcfaabe504
2017-11-01 12:41:38 -04:00
Dave Borowitz f7ceeaa23f FileRepository: Add pack-based inserter implementation
Applications that use ObjectInserters to create lots of individual
objects may prefer to avoid cluttering up the object directory with
loose objects. Add a specialized inserter implementation that produces a
single pack file no matter how many objects. This inserter is loosely
based on the existing DfsInserter implementation, but is simpler since
we don't need to buffer blocks in memory before writing to storage.

An alternative for such applications would be to write out the loose
objects and then repack just those objects later. This operation is not
currently supported with the GC class, which always repacks existing
packs when compacting loose objects. This in turn requires more
CPU-intensive reachability checks and extra I/O to copy objects from old
packs to new packs.

So, the choice was between implementing a new variant of repack, or not
writing loose objects in the first place. The latter approach is likely
less code overall, and avoids unnecessary I/O at runtime.

The current implementation does not yet support newReader() for reading
back objects.

Change-Id: I2074418f4e65853b7113de5eaced3a6b037d1a17
2017-11-01 12:40:53 -04:00
Dave Borowitz 5ce1cc3d43 ObjectDirectory: Factor a method to close open pack handles
Change-Id: I9490c4583fdf801de9c0bb09595ebb8fb4926988
2017-11-01 12:40:13 -04:00
Dave Borowitz 5b5c3a2e1d ObjectDirectory: Remove last modified check in insertPack
GC explicitly handles the case where a new pack has the same name as an
existing pack due to it containing the exact same set of objects. In
this case, the pack passed to insertPack will have the same name as an
existing pack, but it will also almost certainly have a later mtime than
the existing pack.

The loop in insertPack tried to short-circuit when inserting a new pack,
to avoid walking more of the pack list than necessary. Unfortunately,
this means it will never get to the check for an identical name,
resulting in a duplicate entry for the same PackFile in the pack list.
Remove the short-circuit so that insertPack does not insert a duplicate
entry.

Change-Id: I00711b28594622ad3bd104332334e8a3592cda7f
2017-11-01 10:47:59 -04:00
Matthias Sohn abf420302b Reintroduce protected method which removal broke EMF Compare
So far we follow OSGi semantic versioning [1] which says the following:

"A change in the second (minor) part of the version signals that the
change is backward compatible with consumers of the API package but not
with the providers of that API. That is, when the API package goes from
version 1.5 to 1.6 it is no longer compatible with a provider of that
API but consumers of that API are backward compatible with that API
package."

The change Ib5fbf17bdaf727bc5d0e106ce88f2620d9f87a6f broke EMF Compare
which subclasses ResolveMerger since we added a new parameter to the
protected ResolveMerger.processEntry() method. According to the above
cited OSGi semantic versioning this is ok, implementers should expect
that they break on minor version changes of the API they implement.

This change reintroduces the old processEntry() method in order to help
avoid breakage for existing EMF Compare versions which expect breakage
also for the implementer case only for major version change (in this
case from JGit 4.x to 5.x).

[1] http://www.osgi.org/wp-content/uploads/SemanticVersioning1.pdf

See: https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03431.html
Change-Id: I48ba4308dee73925fa32d6c2fd6b5fd89632c571
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-10-24 23:13:37 +02:00
Michael Keppler e1a39cbbe7 Avoid bad rounding "1 year, 12 months" in date formatter
Round first, then calculate the labels. This avoids "x years, 12 months"
and instead produces "x+1 years".

One test case has been added for the original example the bug was found
with, and one assertion has been moved from an existing test case to the
new test case, since it also triggered the bug.

Bug: 525907
Change-Id: I3270af3850c4fb7bae9123a0a6582f93055c9780
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-10-23 11:33:55 +09:00
Thomas Wolf adbf0935e1 Ensure that ~ in ssh config is replaced before Jsch sees it
Do tilde replacement for values from the ssh config file that are
file names in all cases to make sure that they are already replaced
when Jsch tries to get the values.

Previously, OpenSshConfig did tilde replacement only for the
IdentityFile in the JGit-facing "Host" interface and left the
replacement in the Jsch-facing "Config" interface to Jsch.

But on Windows the JGit notion of what should be used to replace the
tilde differs from Jsch's replacement. Jsch always replaces the tilde
by the value of the system property "user.home", whereas JGit also
considers some environment variables like %HOME%. This can lead to
rather surprising failures as in the case of bug 526175 where
%HOME% != user.home.

Prior to commit 9d24470 (i.e.,prior to JGit 4.9.0) this problem never
occurred because Jsch was completely unaware of the ssh config file
and all host and IdentityFile handling happened exclusively in JGit.

Bug: 526175
Change-Id: I1511699664ffea07cb58ed751cfdb79b15e3a99e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-10-18 22:45:36 +02:00
Matthias Sohn 8180183289 Prepare 4.9.1-SNAPSHOT builds
Change-Id: Ic49fd093d3fe4324c4d83aba74033040fcaa37a6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-10-08 11:25:06 +02:00
Matthias Sohn 6877730fa0 JGit v4.9.0.201710071750-r
Change-Id: I487f6aa3d0c4ef1d57f91cdc36177d994ae24c51
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-10-07 23:46:52 +02:00
Till Brychcy 401c6c98b5 Terminate StreamCopy threads in case of errors
- fix NPE: don't invoke close() if no exception happened.

Bug: 513554
Change-Id: I29f9b2ac1607ee26521e8aba334facd20e4ad79c
Signed-off-by: Till Brychcy <register.eclipse@brychcy.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-10-07 00:03:45 +02:00
Zhen Chen 65f9046547 Use a new RevWalk for validating not advertised wants
Shadow commits in the RevWalk in the UploadPack object may cause the
UNINTERESTING flag not being carried over to their parents commits since
they were marked NO_PARENTS during the assumeShallow or
initializeShallowCommits call.

A new RevWalk needs to be created for this reason, but instead of
creating a new RevWalk from Repository, we can reuse the ObjectReader in
the RevWalk of UploadPack to load objects.

Change-Id: Ic3fee0512d35b4f555c60e696a880f8b192e4439
Signed-off-by: Zhen Chen <czhen@google.com>
2017-10-05 17:16:13 -07:00
Dmitry Neverov a0a8a6a3e4 Terminate StreamCopy threads in case of errors
Bug: 513554
Change-Id: I065836b8fc77ade6de92e4a809fcf770a4f793ec
Signed-off-by: Dmitry Neverov <dmitry.neverov@gmail.com>
2017-09-30 14:01:16 +01:00
Dave Borowitz 2bbe15abd4 ReflogWriter: Align auto-creation defaults with C git
Per git-config(1), core.logAllRefUpdates auto-creates reflogs for HEAD
and for refs under heads, notes, tags, and for HEAD. Add notes and
remove stash from ReflogWriter#shouldAutoCreateLog. Explicitly force
writing reflogs for refs/stash at call sites, now that this is
supported.

Change-Id: I3a46d2c2703b7c243e0ee2bbf6948279800c485c
2017-09-30 12:01:19 +01:00
Dave Borowitz 77a28e0d58 Support force writing reflog on a per-update basis
Even if a repository has core.logAllRefUpdates=true, ReflogWriter does
not create reflog files unless the refs are under a hard-coded list of
prefixes, or unless the forceWrite bit is set. Expose the forceWrite bit
on a per-update basis in RefUpdate/BatchRefUpdate/ReceiveCommand,
creating RefLogWriters as necessary.

Change-Id: Ifc851fba00f76bf56d4134f821d0576b37810f80
2017-09-30 11:55:31 +01:00
Dave Borowitz b1ae96bf84 Ensure ReflogWriter only works with a RefDirectory
The ReflogWriter constructor just took a Repository and called
getDirectory() on it to figure out the reflog dirs, but not all
Repository instances use this storage format for reflogs, so it's
incorrect to attempt to use ReflogWriter when there is not a
RefDirectory directly involved. In practice, ReflogWriter was mostly
only used by the implementation of RefDirectory, so enforcing this is
mostly just shuffling around calls in the same internal package.

The one exception is StashDropCommand, which writes to a reflog lock
file directly. This was a reasonable implementation decision, because
there is no general reflog interface in JGit beyond using
(Batch)RefUpdate to write new entries to the reflog. So to implement
"git stash drop <N>", which removes an arbitrary element from the
reflog, it's fair to fall back to the RefDirectory implementation.
Creating and using a more general interface is well beyond the scope of
this change.

That said, the old behavior of writing out the reflog file even if
that's not the reflog format used by the given Repository is clearly
wrong. Fail fast in this case instead.

Change-Id: I9bd4b047bc3e28a5607fd346ec2400dde9151730
2017-09-30 11:54:05 +01:00
Thomas Wolf 6dab29f4b5 HttpConfig: load user config before reading values from it
Same problem as in commit c227268: openUserConfig() just creates the
FileBasedConfig object, but doesn't read the file yet. An explicit
load() is needed.

As HttpConfig is read-only this omission did not cause any bad effects,
but it simply ignored values from the user config. Most uses of
HttpConfig go through the two-argument constructor, though, where
HttpConfig is given an already loaded repo config.

Change-Id: Ibe7c562c17d6ef37de8b661ab7f6fa0246db01a2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-26 23:54:45 +02:00
Thomas Wolf c227268995 Load the user config before modifying it
SystemReader.openUserConfig() does not load the config yet; an
explicit StoredConfig.load() is needed.

Bug: 374703
Change-Id: I1c397e2fb1a07ac4d9de3675d996417734ff90e9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-23 11:15:27 +02:00
David Pursehouse e32aed6d75 SubmoduleUpdateCommand: Add fetch callback
When the submodule already exists, it is fetched instead of
cloned.

Use the fetch callback instead of clone callback in this case.

Change-Id: I170c21ab92b4117f25fdf940fe6807f214b04d39
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-20 21:49:47 +02:00
Michael FIG 06835f3e4f Fetch submodule repo before resolving commits
By default, this is turned off unless cmd.setFetch(true) is given.  It
will default to true in a future release to mimic c-git behaviour.

This is needed to prevent Eclipse from crashing with "Missing unknown
[REF]" when cloning a repo with submodules.

Bug: 470318
Change-Id: I8ae37c7c5bd2408cead8d57dd13e93e01e0e9dc1
Signed-off-by: Michael FIG <michael@fig.org>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-20 21:49:46 +02:00
Thomas Wolf d946f95c9c Handle SSL handshake failures in TransportHttp
When a https connection could not be established because the SSL
handshake was unsuccessful, TransportHttp would unconditionally
throw a TransportException.

Other https clients like web browsers or also some SVN clients
handle this more gracefully. If there's a problem with the server
certificate, they inform the user and give him a possibility to
connect to the server all the same.

In git, this would correspond to dynamically setting http.sslVerify
to false for the server.

Implement this using the CredentialsProvider to inform and ask the
user. We offer three choices:

1. skip SSL verification for the current git operation, or
2. skip SSL verification for the server always from now on for
   requests originating from the current repository, or
3. always skip SSL verification for the server from now on.

For (1), we just suppress SSL verification for the current instance of
TransportHttp.

For (2), we store a http.<uri>.sslVerify = false setting for the
original URI in the repo config.

For (3), we store the http.<uri>.sslVerify setting in the git user
config.

Adapt the SmartClientSmartServerSslTest such that it uses this
mechanism instead of setting http.sslVerify up front.

Improve SimpleHttpServer to enable setting it up also with HTTPS
support in anticipation of an EGit SWTbot UI test verifying that
cloning via HTTPS from a server that has a certificate that doesn't
validate pops up the correct dialog, and that cloning subsequently
proceeds successfully if the user decides to skip SSL verification.

Bug: 374703
Change-Id: Ie1abada9a3d389ad4d8d52c2d5265d2764e3fb0e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-13 23:23:08 +02:00
Thomas Wolf fdcd4f9a34 Support http.<url>.* configs
Git has a rather elaborate mechanism to specify HTTP configuration
options per URL, based on pattern matching the URL against "http"
subsection names.[1] The URLs used for this matching are always the
original URLs; redirected URLs do not participate.

* Scheme and host must match exactly case-insensitively.
* An optional user name must match exactly.
* Ports must match exactly after default ports have been filled in.
* The path of a subsection, if any, must match a segment prefix of
  the path of the URL.
* Matches with user name take precedence over equal-length path
  matches without, but longer path matches are preferred over
  shorter matches with user name.

Implement this for JGit. Factor out the HttpConfig from TransportHttp
and implement the matching and override mechanism.

The set of supported settings is still the same; JGit currently
supports only followRedirects, postBuffer, and sslVerify, plus the
JGit-specific maxRedirects key.

Add tests for path normalization and prefix matching only on segment
separators, and use the new mechanism in SmartClientSmartServerSslTest
to disable sslVerify selectively for only the test server URLs.

Compare also bug 374703 and bug 465492. With this commit it would be
possible to set sslVerify to false for only the git server using a
self-signed certificate instead of having to switch it off globally
via http.sslVerify.

[1] https://git-scm.com/docs/git-config

Change-Id: I42a3c2399cb937cd7884116a2a32fcaa7a418fcb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-10 17:37:54 -04:00
David Pursehouse 2dbfe49a42 Add PushConfig class with PushRecurseSubmodulesMode
This will be used later when adding for support for recursing
submodules on push.

Change-Id: Ie2a183e5404a32046de9f6524e6ceeec37919671
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-09 13:58:09 -04:00
Thomas Wolf c27f36dfc7 Fix missing RefsChangedEvent when packed refs are used
With atomic ref updates using packed refs, JGit did not fire a
RefsChangedEvent. This resulted in a user-visible regression in
EGit: the UI would not update after a "Fetch from upstream...".
Presumably it would also make Gerrit miss out on ref changes?

Strengthen the BatchRefUpdateTest by also asserting the expected
number of RefsChangedEvents, and ensure modCnt is incremented in
RefDirectory.commitPackedRefs() when refs really changed (as opposed
to some internal housekeeping operation, such as packing loose refs).

Bug: 521296
Change-Id: Ia985bda1d99f45a5f89c8020ca4845e7a66e743e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-07 18:47:45 -04:00
Dave Borowitz bb09e09344 Add FetchCommand#setRefSpecs(String...) variant
Much of the time the caller can specify a RefSpec succinctly using a
string, and doesn't care about calling setters. Add a convenience method
for this case, and use it where applicable in JGit core.

Change-Id: Ic3fac7fc568eee4759236a5264d2e7e5f9b9716d
2017-09-07 07:46:25 -04:00
Shawn Pearce e68a9b3ed8 ReceivePack: clear advertised .haves if application changes refs
An application can choose to invoke setAdvertisedRefs multiple times,
for example several AdvertiseRefsHook installed in a chain. Each of
these invocations populates the advertisedHaves collection with the
unique set of ObjectIds.

This can lead to a server over-advertising with ".have" lines if the
first hook pushes in a lot of references, and the second hook filters
this to a subset.  ReceivePack will advertise the unique objects from
the first hook using ".have" lines, which may lead to a huge
advertisement sent to the client.

This can also contribute to a very slow connectivity check after the
pack is parsed as ReceivePack calls markUninteresting on every commit
in advertisedHaves.  This may require expanding a lot of subtrees to
mark all trees as uninteresting as well.  On a very big repository
this can lead to a many-second stall.

Clear the advertisedHaves collection any time the refs are updated.
Add a test to verify the correct set of objects was sent.

Change-Id: I97f6998d0597251444a2e846a3ea1f461bae96f9
2017-09-07 05:39:47 -04:00
Shawn Pearce d0d15c3848 DfsGarbageCollector: support disabling conversion to reftable
When a repository is initially created using only reftable but doesn't
yet have a GC pack, the garbage collector shouldn't scan the ref
database. Support disabling the reftable conversion path.

Change-Id: Iaaf3a4375cd43760b7181c1bd10244de8b2c5d9e
2017-09-06 16:37:54 -07:00
Shawn Pearce f0a40b1933 Remove dead warning about minUpdaeIndex and maxUpdateIndex affecting refresh
DfsGarbageCollector always performs refreshes.  This warning was from
a prior iteration of the patch set and should have been removed before
the change was merged.

Change-Id: Id4750bb6c1c177e1ef040fd22912f434bbb5e992
2017-09-06 11:12:34 -07:00
Matthias Sohn 5bd9392f96 Remove no longer needed API problem filters
Change-Id: I206682a60ad9228c89efa22f8db9e72e19f17f6d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-09-06 00:58:09 +02:00
David Pursehouse f9ac86fde2 SubmoduleConfig: Simplify #toConfigValue
Just return configValue instead of transforming name().

Change-Id: I7f94ab2e206e93d1370467e187c5e68e8f6a8836
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-05 23:30:17 +02:00
Shawn Pearce d26309c4d9 Merge changes from topic 'dfs-reftable'
* changes:
  dfs: write reftable from DfsGarbageCollector
  dfs: compact reftables during DfsPackCompactor
2017-09-05 12:39:53 -04:00
Shawn Pearce d13dfac9dc dfs: write reftable from DfsGarbageCollector
If a ReftableConfig has been supplied by the caller, write out a
reftable as a sibling of the the GC pack, alongside the heads.

To bootstrap from a non-reftable system, the refs are read from the
DfsRefDatabase if no GC reftables are present.  Its assumed the
references are fully current, and do not need to be merged with any
other reftables.  Any non-GC reftables will be pruned at the end of
the GC cycle, just like any packs that were replaced.

If a GC reftable is present, all existing reftables are compacted, and
references from DfsRefDatabase are only used to seed the packer.  Its
assumed these are consistent with each other.

Change-Id: Ie397eb58aaaefb6865c816d9b39de3ac12998019
2017-09-05 09:10:16 -07:00
Shawn Pearce d126bcc5c8 dfs: compact reftables during DfsPackCompactor
Combine intermediate, non-GC reftables when combining pack files.
This shrinks the reftable stack, improving lookup times.

Change-Id: I5dbba41806f99af5ecaff3a3119f6630e9404256
2017-09-05 09:05:08 -07:00
Thomas Wolf 11c476346d Fix Daemon.stop() to actually stop the listener thread
ServerSocket.accept() is not interruptible: a thread busy in accept()
may not react to Thread.interrupt() and may not return from accept()
via an InterruptedException. Close the socket instead to make the
daemon's listener thread terminate.

* Close the listening socket to get the listening thread to exit
  instead of interrupting it.
* Add a stopAndWait() method that stops the listening thread and
  then waits until it has indeed finished.
* Set SO_REUSE_ADDRESS on the listening socket.

Bug: 376369
Change-Id: I9d6014103e6dcb0173daea134feb44dc52c5c69a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-04 23:50:48 +02:00
Shawn Pearce de4e0acc30 Merge "Remove workaround for bug in Java's ReferenceQueue" 2017-09-04 14:35:58 -04:00
Matthias Sohn 39b193b6f4 Remove workaround for bug in Java's ReferenceQueue
Sun's Java 5, 6, 7 implementation had a bug [1] where a Reference can be
enqueued and dequeued twice on the same reference queue due to a race
condition within ReferenceQueue.enqueue(Reference).

This bug was fixed for Java 8 [2] hence remove the workaround.

[1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6837858
[2] http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/858c75eb83b5

Change-Id: I2deeb607e3d237f9f825a207533acdee305c7e73
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-09-04 14:44:12 +02:00
Thomas Wolf 6d27869659 Use Config.getEnum() to read the IgnoreSubmoduleMode
Doing so goes through the TypedConfigGetter and thus allows library
clients (for instance EGit) to warn about invalid configurations.

Change-Id: If1080ad90b8aff54a903d4d75637614faad6469b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-09-04 09:19:03 +02:00
Thomas Wolf 06ea633c18 Don't assume name = path in .gitmodules
While parsing .gitmodules, the name of the submodule subsection is
purely arbitrary: it frequently is the path of the submodule, but
there's no requirement for it to be. By building a map of paths to
the section name in .gitmodules, we can more accurately return
the submodule URL.

Bug: 508801
Change-Id: I8399ccada1834d4cc5d023344b97dcf8d5869b16
Also-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-04 09:19:03 +02:00
David Pursehouse 53deba1b4f FsckPackParser: Fix warning about int being boxed to Integer
Change-Id: I525164090e3990ce353bbb1627aaafaad2c903b6
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-09-01 00:14:32 +09:00
David Pursehouse de5eb5e2cd FsckPackParser: Fix "Empty block should be documented" warnings
Add "Do nothing" comments, consistent with other empty methods in
the same class.

Change-Id: I27a13a402e94104af617be0e14d8982e75fa73bd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
David Pursehouse 9c1fcf6ad7 FsckPackParser: Fix warnings about variable hiding
Change-Id: Id1ada1cedfad5671e79df303c6dbcb6017b777ce
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
David Pursehouse f8c2067410 Silence API errors caused by adding enum constants in 1baf86d4d2
Change-Id: I23338c04567f95aedd909ab310e01277e8f7db69
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-31 21:40:10 +09:00
Shawn Pearce f4329b09d4 Merge changes from topic 'dfs-reftable'
* changes:
  dfs: helper to open multiple reftables
  dfs: expose DfsReftable from DfsObjDatabase
  dfs: support reading reftables through DfsBlockCache
2017-08-30 20:42:23 -04:00
Shawn Pearce 0ca1fb0fe0 Merge changes from topic 'fsck'
* changes:
  DfsFsck: reduce memory usage during verifyIndex
  DfsFsck: refactor pack verify into its own method
  DfsFsck: run connectivity check pass exactly once
2017-08-30 08:42:37 -04:00
Shawn Pearce e5db7c1f0e DfsFsck: reduce memory usage during verifyIndex
Don't convert a lot of ObjectId to String stored in generic
java.util.HashSet.  This is a very expensive way to store objects.

Instead rely on "this" from the FsckPackParser to lookup information
about the objects in this pack file, which lets the verify code avoid
sorting the object list.

Use ObjectIdOwnerMap, which is the most efficient format JGit has
for storing lots of objects.

Change-Id: Ib68f93acb4d91b96d0a44c0612f704500d332ac1
2017-08-30 03:11:53 -07:00
Shawn Pearce 69588c21fe DfsFsck: refactor pack verify into its own method
This simplifies the logic about allocation of the DfsReader, and
clarifies the code considerably by using smaller scopes with less
indentation.

A few static imports from PackExt and slightly shorter variable names
make for a more understandable-at-glance implementation.

Change-Id: Iaf5a0e14fe0349215d9e44446f68d1129ad3bb3d
2017-08-30 03:11:43 -07:00
Shawn Pearce 153c11a49b DfsFsck: run connectivity check pass exactly once
The simpler algorithm is to load all branch tips into an ObjectWalk
and run that walk exactly once.  This avoids redoing work related to
parsing and considering trees reused across side branches.

Move the connectivity check into its own helper method.  This moves it
left one level of identation, and makes it easier to fit the method's
logic with less line wrapping.

Add a "Counting objects..." progress monitor around this phase.  Its
what is used when a server receives a push and is also trying to
verify the client sent all required objects.

Change-Id: I4d53d75d0cdd1a13fff7d513a6ae0b2d14ea4090
2017-08-29 18:14:51 -07:00
Matthias Sohn f4c5a54aa9 Cleanup: Remove redundant type arguments
Change-Id: I81ce2e25666f5acfce4d3ada6b71ae8c81a97a4d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:34 +02:00
Matthias Sohn e9fb111182 Cleanup: Organize imports
Change-Id: I6065e59519bc42bd18f5cc5ee3ec5210764ab03c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:33 +02:00
Matthias Sohn 41baa7a791 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Do not automatically organize imports using a save action since this
seems to be buggy and removed some annotations org.eclipse.jgit.pgm
needs to use args4j.

Change-Id: I5a91292c3b9241ce2dde3e4ecce14ad460097129
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 03:07:18 +02:00
Matthias Sohn 1052678372 Cleanup: Remove trailing whitespace
Change-Id: I7c08b60fb6afafd1874210320da3da7519632e08
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 01:24:48 +02:00
Matthias Sohn 9906f09868 Partially revert c0ad77d8 "Enhance Eclipse save actions"
Revert the following save actions which were introduced in c0ad77d8:
- always use braces around blocks
- remove unused imports

Other than I expected save actions are run globally on edited files -
and not only on edited code lines only.

Hence revert the save action "Convert control statement bodies to
blocks" which would affect a large number of code lines not affected by
the change editing some small part of a class. This would generate a
large number of changes which may lead to many unnecessary conflicts.
Total number of affected lines across jgit would be around 10k lines.

Also revert "Remove unused imports" since it erroneously removes imports
of some annotations needed by pgm classes using args4j.

Change-Id: I879a47f68e664129e6124cf25c1ae1f6a2d7a5aa
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-30 01:24:47 +02:00
Thomas Wolf dbef8e2537 Pass along the original exception when an ssh connection fails
Otherwise, the stack trace doesn't really tell anything.
See for instance [1].

[1] https://www.eclipse.org/forums/index.php/t/1088535/

Change-Id: If22f2c63c36fec6b32818d2c2acecf20531b4185
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-29 15:08:08 -04:00
Shawn Pearce 4f7ce2b46b dfs: helper to open multiple reftables
ReftableStack maintains multiple open reftables
in an AutoCloseable format, making it easier for
higher level code to handle multiple files.

Change-Id: I7ac35c18e67b7e771fb3de29169d1ee50fab62ca
2017-08-28 15:07:05 -07:00
Shawn Pearce 1a7b8a11df dfs: expose DfsReftable from DfsObjDatabase
Reftable storage in DFS is related to pack storage.  Reftables are
stored in the same namespace, but with PackExt.REFTABLE.  Include
the set of DfsReftable instances in the PackList and export some
helpers to access the tables.

Change-Id: I6a4f5f953ed6b0ff80a7780f4c6cbcc5eda0da3e
2017-08-28 15:07:04 -07:00
Shawn Pearce 1222f34506 dfs: support reading reftables through DfsBlockCache
DfsBlockCache directly shares its internal byte[] with ReftableReader,
avoding copying between the DfsBlockCache and the BlockReader
instances used by ReftableReader.

Change-Id: Icaa4f40052b26f952681414653a8b5314b7c2c23
2017-08-28 15:07:04 -07:00
Shawn Pearce d684ade3d3 Merge "reftable: explicitly store update_index per ref" 2017-08-28 17:57:13 -04:00
David Pursehouse 10a8df22fa Merge "Enhance Eclipse save actions" 2017-08-28 12:08:01 -04:00
Terry Parker cb24de07d0 Merge "Add BlobObjectChecker" 2017-08-28 12:00:53 -04:00
Matthias Sohn c0ad77d84c Enhance Eclipse save actions
Add the following Eclipse save actions executed when saving modified
lines. This should help to reduce manual work needed to maintain a clean
and consistent code style:
- organize imports
- always use braces around blocks
- add missing annotations
  - @Override including implementation of interface methods
  - @Deprecated
- remove
  - unused imports
  - unnecessary $NON-NLS$ tags
  - redundant type arguments

Also add default values for new settings that were introduced in recent
Eclipse versions up to Neon since we updated save rules the last time.

Change-Id: Idc90b249df044d0552f04edf01a5f607c4846f50
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-28 11:52:45 -04:00
Masaya Suzuki fd74cf2f78 Add BlobObjectChecker
Some repositories can have a policy that do not accept certain blobs. To
check if the incoming pack file contains such blobs, ObjectChecker can
be used. However, this ObjectChecker is not called by PackParser if the
blob is stored as a whole. This is because the object can be so large
that it doesn't fit in memory.

This change introduces BlobObjectChecker. This interface takes chunks of
a blob instead of the entire object. ObjectChecker can optionally return
a BlobObjectChecker. This won't change existing ObjectChecker
implementation; existing implementation continues to receive deltified
blob objects only.

Change-Id: Ic33a92c2de42bd7a89786a4da26b7a648b25218d
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2017-08-28 08:42:27 -07:00
Thomas Wolf 1637c44048 FetchCommand: pass on CredentialsProvider to submodule fetches
When a JGit API command is implemented in terms of other API
commands, the child command must "inherit" all relevant settings.
Calling configure() ensures that the CredentialsProvider and the
connection timeout are propagated correctly.

Bug: 515325
Change-Id: I948e306693a9edb7b199a735877413b6eddcfba4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:37:43 +02:00
Thomas Wolf d031b64667 Exclude file matching: fix backtracking on match failures after **
** matching always tries the empty match first. If a mismatch occurs
later, the ** must be extended by exactly one segment and matching must
resume with the matcher following the ** matcher.

Bug: 520920
Change-Id: Id019ad1c773bd645ae92e398021952f8e961f45c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:41 +02:00
Thomas Wolf d80b999c76 Fix path pattern matching to work also for gitattributes
Path pattern matching for attribute rules is different than matching
for excluded files.

The first difference concerns patterns without slashes. For
gitattributes those must match on the last component only, not on
any earlier segment. This is true also for directory-only patterns.

The second difference concerns directory-only patterns. Those also
must not match on a prefix or segment except the last one. They do
not apply recursively to all files beneath.

And third, matches only on a prefix must match for gitattributes
only if the last matcher was "/**".

Add a new parameter for such path matching to IMatcher.matches() and
pass it through as appropriate (false for gitignore, true for
gitattributes). As far as gitignore is concerned, there is no change.

New tests have been added, and some existing attribute matching tests
have been fixed since they operated on wrong assumptions.

Bug: 508568
Change-Id: Ie825dc2cac8a85a72a7eeb0abb888f3193d21dd2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-27 16:02:40 +02:00
Thomas Wolf 426caf99ee Ignore invalid TagOpt values
C git silently ignores invalid tagopt values; so make JGit behave the
same way.

Bug: 429625
Change-Id: I99587cc46c7e0c19348bcc63f602038fa9a7f378
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 09:11:03 +02:00
Thomas Wolf 8cbdf523cd Add a getter for a list of RefSpecs to Config
Reading RefSpecs from a Config can be seen as another typed value
conversion, so add a getter to Config and to TypedConfigGetter. Use
it in RemoteConfig.

Doing this allows clients of the JGit library to customize the
handling of invalid RefSpecs in git config files by installing a
custom TypedConfigGetter.

Bug: 517314
Change-Id: I0ebc0f073fabc85c2a693b43f5ba5962d8a795ff
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 09:11:02 +02:00
Thomas Wolf d32ad1cadd Improve getting typed values from a Config
Make the handling of typed values somewhat configurable by using
a separate converter. The default converter is the same as before;
just the implementations of the getters were moved. They also still
raise IllegalArgumentException on invalid values as before.

The converter can be set globally via Config.setTypedConfigGetter(),
which EGit can use in its core Activator to plug in a variant that
catches the IllegalArgumentException, logs the problem, and then
returns the default value.

In this way the behavior for other users of the JGit library is
unchanged, while EGit can deal gracefully with invalid git configs.

Bug: 520978
Change-Id: Ie8f81d206e358b6cc57aa29b9d7ad2a5d34b86a1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 09:11:02 +02:00
Matthias Sohn 960d7ff3e5 Prepare 4.5.4-SNAPSHOT builds
Change-Id: Id8b902bf2bf590b41f2e246c5ecf1592e1c411f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 08:08:46 +02:00
David Pursehouse e237c28936 Merge "Fix JGit set core.fileMode to false by default instead of true for non Windows OS." 2017-08-25 20:58:07 -04:00
David Pursehouse 40f40e496a Merge "Fix default directory set when setDirectory wasn't called." 2017-08-25 20:57:52 -04:00
David Pursehouse 0e12692d8c FileMode: Remove unnecessary @SuppressWarnings("synthetic-access")
In Eclipse Oxygen, the following warning is emitted:

  At least one of the problems in category 'synthetic-access' is not
  analysed due to a compiler option being ignored

Removing the suppression gets rid of the warning.

Change-Id: Ibfe5cc1e347150b699f54e2f204ab5ee770da202
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-25 20:57:11 -04:00
Matthias Sohn d979dfd00c Add toString() methods to OpenSshConfig to help debugging
Change-Id: I81b60a13a97e78d5ccd593ba8e4aa614df19f925
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 01:44:37 +02:00
Thomas Wolf c758a8cd37 Do most %-token substitutions in OpenSshConfig
Except for %p and %r and partially %C, we can do token substitutions
as defined by OpenSSH inside the config file parser. %p and %r can
be replaced only if specified in the config; if not, it would be the
caller's responsibility to replace them with values obtained from the
URI to connect to.

Jsch doesn't know about token substitutions at all. By doing the
replacements as good as we can in the config file parser, we can
make Jsch support most of these tokens.

%i is not handled at all as Java has no concept of a "user ID".

Includes unit tests.

Bug: 496170
Change-Id: If9d324090707de5d50c740b0d4455aefa8db46ee
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-26 01:44:36 +02:00
Thomas Wolf 9d2447063d Let Jsch know about ~/.ssh/config
Ensure the Jsch instance used knows about ~/.ssh/config. This
enables Jsch to honor more user configurations (see
com.jcraft.jsch.Session.applyConfig()), in particular also the
UserKnownHostsFile configuration, or additional identities given
via multiple IdentityFile entries.

Turn JGit's OpenSshConfig into a full parser that can be a
Jsch-compliant ConfigRepository. This avoids a few bugs
in Jsch's OpenSSHConfig and keeps the JGit-facing interface
unchanged. At the same time we can supply a JGit OpenSshConfig
instance as a ConfigRepository to Jsch. And since they'll both
work from the same object, we can also be sure that the parsing
behavior is identical.

The parser does not handle the "Match" and "Include" keys, and it
doesn't do %-token substitutions (yet).

Note that Jsch doesn't handle multi-valued UserKnownHostFile
entries as known by modern OpenSSH.[1]

[1] http://man.openbsd.org/OpenBSD-current/man5/ssh_config.5

Additional tests for new features are provided in OpenSshConfigTest.

Bug: 490939
Change-Id: Ic683bd412fa8c5632142aebba4a07fad4c64c637
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-26 01:41:50 +02:00
Masaya Suzuki 9fb6561e7a Consume request body before flushing the buffer
This is continuation from https://git.eclipse.org/r/#/c/94249/. When an
error happens, we might not read the entire stream. Consume the request
body before we flush the buffer.

Change-Id: Ia473a04ace600653b2d1f2822e3023570d992410
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
2017-08-25 15:23:20 -07:00
Joan Goyeau 88e453995d Fix default directory set when setDirectory wasn't called.
Bug: 519883
Change-Id: I46716e9626b4c4adc0806a7c8df6914309040b94
Signed-off-by: Joan Goyeau <joan@goyeau.com>
2017-08-25 11:41:40 +01:00
David Pursehouse 65b2d0b2d9 ObjectToPack: Add missing @Override annotation
Change-Id: I65ed7b89312d58ea816b46d27707ff907df1c78b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
2017-08-24 16:20:11 +09:00
Thomas Wolf 1b4daa2994 Cleanup: message reporting for HTTP redirect handling
The addition of "tooManyRedirects" in commit 7ac1bfc ("Do
authentication re-tries on HTTP POST") was an error I didn't
catch after rebasing that change. That message had been renamed
in the earlier commit e17bfc9 ("Add support to follow HTTP
redirects") to "redirectLimitExceeded".

Also make sure we always use the TransportException(URIish, ...)
constructor; it'll prefix the message given with the sanitized URI.
Change messages to remove the explicit mention of that URI inside the
message. Adapt tests that check the expected exception message text.

For the info logging of redirects, remove a potentially present
password component in the URI to avoid leaking it into the log.

Change-Id: I517112404757a9a947e92aaace743c6541dce6aa
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-23 12:20:55 +02:00
Thomas Wolf 7ac1bfc834 Do authentication re-tries on HTTP POST
There is at least one git server out there (GOGS) that does
not require authentication on the initial GET for
info/refs?service=git-receive-pack but that _does_ require
authentication for the subsequent POST to actually do the push.

This occurs on GOGS with public repositories; for private
repositories it wants authentication up front.

Handle this behavior by adding 401 handling to our POST request.
Note that this is suboptimal; we'll re-send the push data at
least twice if an authentication failure on POST occurs. It
would be much better if the server required authentication
up-front in the GET request.

Added authentication unit tests (using BASIC auth) to the
SmartClientSmartServerTest:

- clone with authentication
- clone with authentication but lacking CredentialsProvider
- clone with authentication and wrong password
- clone with authentication after redirect
- clone with authentication only on POST, but not on GET

Also tested manually in the wild using repositories at try.gogs.io.
That server offers only BASIC auth, so the other paths
(DIGEST, NEGOTIATE, fall back from DIGEST to BASIC) are untested
and I have no way to test them.

* public repository: GET unauthenticated, POST authenticated
  Also tested after clearing the credentials and then entering a
  wrong password: correctly asks three times during the HTTP
  POST for user name and password, then gives up.
* private repository: authentication already on GET; then gets
  applied correctly initially to the POST request, which succeeds.

Also fix the authentication to use the credentials for the redirected
URI if redirects had occurred. We must not present the credentials
for the original URI in that case. Consider a malicious redirect A->B:
this would allow server B to harvest the user credentials for server
A. The unit test for authentication after a redirect also tests for
this.

Bug: 513043
Change-Id: I97ee5058569efa1545a6c6f6edfd2b357c40592a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-22 23:57:09 +02:00
Shawn Pearce 44a75d9ea8 reftable: explicitly store update_index per ref
Add an update_index to every reference in a reftable, storing the
exact transaction that last modified the reference.  This is necessary
to fix some merge race conditions.

Consider updates at T1, T3 are present in two reftables.  Compacting
these will create a table with range [T1,T3].  If T2 arrives during
or after the compaction its impossible for readers to know how to
merge the [T1,T3] table with the T2 table.

With an explicit update_index per reference, MergedReftable is able to
individually sort each reference, merging individual entries at T3
from [T1,T3] ahead of identically named entries appearing in T2.

Change-Id: Ie4065d4176a5a0207dcab9696ae05d086e042140
2017-08-21 15:39:08 -07:00
Shawn Pearce 2d76df2442 reftable: reserve standard PackExt
Reserve "ref" extension for reftable files.  This allows them to be
used in a DFS repository as a stream in a DfsPackDescription.

Change-Id: Ife781bb64d0bb063333183ad2be70a41a2482513
2017-08-17 15:06:51 -07:00
Shawn Pearce 0aae64ce74 reftable: resolve symbolic references
resolve(Ref) helps callers recursively chase symbolic references and
is a useful function when wrapping a Reftable inside a RefDatabase, as
RefCursor does not resolve symbolic references during iteration.

Change-Id: I1ba143f403773497972e225dc92c35ecb989e154
2017-08-17 15:06:51 -07:00
Shawn Pearce 195541dd30 reftable: support threshold based compaction
Transactions may wish to merge several tables together as part of an
operation.  Setting a byte limit allows the transaction to consider
only some recent tables, bounding the cost of the compaction.

Change-Id: If037f2cbdc174ff1a215d5917178b33cde4ddaba
2017-08-17 15:06:51 -07:00
Shawn Pearce d48ac5bf01 reftable: compact merged tables
A compaction of reftables is just copying the results of a
MergedReftable into a ReftableWriter.  Wrap this up into a utility.

Change-Id: I6f5677d923e9628993a2d8b4b007a9b8662c9045
2017-08-17 15:06:51 -07:00
Shawn Pearce 77d8eead6d reftable: merge-join reftables
MergedReftable combines multiple reference tables together in a stack,
allowing higher/later tables to shadow earlier/lower tables.  This
forms the basis of a transaction system, where each transaction writes
a new reftable containing only the modified references, and readers
perform a merge on the fly to get the latest value.

Change-Id: Ic2cb750141e8c61a8b2726b2eb95195acb6ddc83
2017-08-17 15:06:51 -07:00
Shawn Pearce 0398f3dd6e reftable: debug tools
Simple debug programs to experiment with the reftable file format:

  debug-read-reftable
  debug-write-reftable
  debug-verify-reftable
  debug-benchmark-reftable

Change-Id: I79db351d86900f1e58b17e922e195dff06ee71f1
2017-08-17 15:06:51 -07:00
Shawn Pearce 0a26dcf4a3 reftable: scan and lookup reftable files
ReftableReader provides sequential scanning support over all
references, a range of references within a subtree (such as
"refs/heads/"), and lookup of a single reference.  Reads can be
accelerated by an index block, if it was created by the writer.

The BlockSource interface provides an abstraction to read from the
reftable's backing storage, supporting a future commit to connect
to JGit DFS and the DfsBlockCache.

Change-Id: Ib0dc5fa937d0c735f2a9ff4439d55c457fea7aa8
2017-08-17 15:06:51 -07:00
Shawn Pearce 0ecc8367e6 reftable: create and write reftable files
This is a simple writer to create reftable formatted files.  Follow-up
commits will add support for reading from reftable, debugging
utilities, and tests.

Change-Id: I3d520c3515c580144490b0b45433ea175a3e6e11
2017-08-17 15:06:50 -07:00
Thomas Wolf e17bfc96f2 Add support to follow HTTP redirects
git-core follows HTTP redirects so JGit should also provide this.

Implement config setting http.followRedirects with possible values
"false" (= never), "true" (= always), and "initial" (only on GET, but
not on POST).[1]

We must do our own redirect handling and cannot rely on the support
that the underlying real connection may offer. At least the JDK's
HttpURLConnection has two features that get in the way:

* it does not allow cross-protocol redirects and thus fails on
  http->https redirects (for instance, on Github).
* it translates a redirect after a POST to a GET unless the system
  property "http.strictPostRedirect" is set to true. We don't want
  to manipulate that system setting nor require it.

Additionally, git has its own rules about what redirects it accepts;[2]
for instance, it does not allow a redirect that adds query arguments.

We handle response codes 301, 302, 303, and 307 as per RFC 2616.[3]
On POST we do not handle 303, and we follow redirects only if
http.followRedirects == true.

Redirects are followed only a certain number of times. There are two
ways to control that limit:

* by default, the limit is given by the http.maxRedirects system
  property that is also used by the JDK. If the system property is
  not set, the default is 5. (This is much lower than the JDK default
  of 20, but I don't see the value of following so many redirects.)
* this can be overwritten by a http.maxRedirects git config setting.

The JGit http.* git config settings are currently all global; JGit has
no support yet for URI-specific settings "http.<pattern>.name". Adding
support for that is well beyond the scope of this change.

Like git-core, we log every redirect attempt (LOG.info) so that users
may know about the redirection having occurred.

Extends the test framework to configure an AppServer with HTTPS support
so that we can test cloning via HTTPS and redirections involving HTTPS.

[1] https://git-scm.com/docs/git-config
[2] 6628eb41db
[3] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

CQ: 13987
Bug: 465167
Change-Id: I86518cb76842f7d326b51f8715e3bbf8ada89859
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-17 22:16:44 +02:00
Christian Halstrick be767fd7d9 Merge "Fix off-by-one error in Strings.count()" 2017-08-16 06:24:43 -04:00
Christian Halstrick c71af0c73a Merge "Use relative paths for attribute rule matching" 2017-08-16 06:24:33 -04:00
Matthias Sohn e21e2436d3 JGit v4.5.3.201708160445-r
Change-Id: I2d57144976e3683e180d3a42edc6c3bf2905e87c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-16 10:42:27 +02:00
Thomas Wolf b13a285098 Send a detailed event on working tree modifications
Currently there is no way to determine the precise changes done
to the working tree by a JGit command. Only the CheckoutCommand
actually provides access to the lists of modified, deleted, and
to-be-deleted files, but those lists may be inaccurate (since they
are determined up-front before the working tree is modified) if
the actual checkout then fails halfway through. Moreover, other
JGit commands that modify the working tree do not offer any way to
figure out which files were changed.

This poses problems for EGit, which may need to refresh parts of the
Eclipse workspace when JGit has done java.io file operations.

Provide the foundations for better file change tracking: the working
tree is modified exclusively in DirCacheCheckout. Make it emit a new
type of RepositoryEvent that lists all files that were modified or
deleted, even if the checkout failed halfway through. We update the
'updated' and 'removed' lists determined up-front in case of file
system problems to reflect the actual state of changes made.

EGit thus can register a listener for these events and then knows
exactly which parts of the Eclipse workspace may need to be refreshed.

Two commands manage checking out individual DirCacheEntries themselves:
checkout specific paths, and applying a stash with untracked files.
Make those two also emit such a new WorkingTreeModifiedEvent.

Furthermore, merges may modify files, and clean, rm, and stash create
may delete files.

CQ: 13969
Bug: 500106
Change-Id: I7a100aee315791fa1201f43bbad61fbae60b35cb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-15 16:52:00 -04:00
Matthias Sohn 81d020aba9 Merge branch 'stable-4.8'
* stable-4.8:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: Ica20f5aa0d8a365fe3317765b93520b3abd5d342
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-15 00:48:44 +02:00
Matthias Sohn 758a181b82 Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: I1e4fcf84506ff4316567bbb1713e84d8d196c2a1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-15 00:24:49 +02:00
Matthias Sohn 53becf1f59 Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
  Update Oxygen Orbit p2 repository to R20170516192513
  Fix exception handling for opening bitmap index files

Change-Id: I669fe48ce0034f9ea1977d38ee39099497422c1c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 23:50:52 +02:00
Matthias Sohn 985e3c6414 Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
  Fix exception handling for opening bitmap index files

Change-Id: Ifb511238e3e98b1bc9f79a990807b940a17ebaa6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 23:43:05 +02:00
Christian Halstrick 1ed1e40387 Fix exception handling for opening bitmap index files
When creating a new PackFile instance it is specified whether this pack
has an associated bitmap index file or not. This information is cached
and the public method getBitmapIndex() will always assume a bitmap index
file must exist if the cached data tells so. But it may happen that the
packfiles are repacked during a gc in a different process causing the
packfile, bitmap-index and index file to be deleted. Since JGit still
has an open FileHandle on the packfile this file is not really deleted
and can still be accessed. But index and bitmap index file are deleted.
Fix getBitmapIndex() to invalidate the cached packfile instance if such
a situation occurs.

This problem showed up when a gerrit server was serving repositories
which where garbage collected with native git regularly. Fetch and
clone commands for certain repositories failed permanently after a
native git gc had deleted old bitmap index files.

Change-Id: I8e620bec74dd3f310ba42024f9a657062f868f0e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
2017-08-14 21:09:48 +02:00
Thomas Wolf 37908321c0 Do not apply pushInsteadOf to existing pushUris
Per the git config documentation[1], pushInsteadOf is ignored when
a remote has explicit pushUris.

Implement this, and adapt tests.

Up to now JGit mistakenly applied pushInsteadOf also to existing
pushUris. If some repositories had relied on this mis-feature,
pushes may newly suddenly fail (the uncritical case; the config
just needs to be fixed) or even still succeed but push to unexpected
places, namely to the non-rewritten pushUrls (the critical case).

The release notes should point out this change.

[1] https://git-scm.com/docs/git-config

Bug: 393170
Change-Id: I38c83204d2ac74f88f3d22d0550bf5ff7ee86daf
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 17:27:05 +02:00
Thomas Wolf df3469f6ad Record submodule paths with untracked changes as FileMode.GITLINK
Bug: 520702
Change-Id: I9bb48af9e8f1f2ce7968a82297c7c16f1237f987
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 14:03:51 +02:00
Thomas Wolf f5a2c77dc4 Fix handling of pushInsteadOf
According to [1], pushInsteadOf is

1. applied to the uris, not to the pushUris
2. ignored if a remote has an explicit pushUri

JGit applied it only to the pushUris. As a result, pushInsteadOf was
ignored for remotes having only a uri, but no pushUri.

This commit implements (1) if there are no pushUris. I did not dare
implement (2) because:

* there are explicit tests for it that expect that pushInsteadOf gets
  applied to existing pushUrls, and
* people may actually use and rely on this JGit behavior.

[1] https://git-scm.com/docs/git-config

Bug: 393170
Change-Id: I6dacbf1768a105190c2a8c5272e7880c1c9c943a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 05:40:47 -04:00
Christian Halstrick 196915dde5 Merge "Ensure EOL stream type is DIRECT when -text attribute is present" 2017-08-14 03:34:57 -04:00
Thomas Wolf b07db60908 Fix off-by-one error in Strings.count()
Change-Id: I0667b1624827d1cf0cc1b81f86c7bb44eafd68a7
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-14 08:04:56 +02:00
Shawn Pearce 53dd9a9e4b Rename extensions.refsStorage to refStorage
This matches the proposal that has been discussed at length on
git-core mailing list and seems to be the accepted convention.

Change-Id: I9f6ab15144826893d1e2a4b48a2d657d6dd445ec
2017-08-11 18:20:50 -07:00
Thomas Wolf a489a8ae9a Ensure EOL stream type is DIRECT when -text attribute is present
Otherwise fancy combinations of attributes (binary or -text in
combination with crlf or eol) may result in the corruption of binary
data.

Bug: 520910
Change-Id: I3ffc666c13d1b9d2ed987b69a67bfc7f42ccdbfc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
2017-08-11 22:56:50 +02:00