90 lines
4.2 KiB
Plaintext
90 lines
4.2 KiB
Plaintext
|
JGit Storage on DHT
|
||
|
|
-------------------
|
||
|
|
|
||
|
|
This implementation still has some pending issues:
|
||
|
|
|
||
|
|
* DhtInserter must skip existing objects
|
||
|
|
|
||
|
|
DirCache writes all trees to the ObjectInserter, letting the
|
||
|
|
inserter figure out which trees we already have, and which are new.
|
||
|
|
DhtInserter should buffer trees into a chunk, then before writing
|
||
|
|
the chunk to the DHT do a batch lookup to find the existing
|
||
|
|
ObjectInfo (if any). If any exist, the chunk should be compacted to
|
||
|
|
eliminate these objects, and if there is room in the chunk for more
|
||
|
|
objects, it should go back to the DhtInserter to be filled further
|
||
|
|
before flushing.
|
||
|
|
|
||
|
|
This implies the DhtInserter needs to work on multiple chunks at
|
||
|
|
once, and may need to combine chunks together when there is more
|
||
|
|
than one partial chunk.
|
||
|
|
|
||
|
|
* DhtPackParser must check for collisions
|
||
|
|
|
||
|
|
Because ChunkCache blindly assumes any copy of an object is an OK
|
||
|
|
copy of an object, DhtPackParser needs to validate all new objects
|
||
|
|
at the end of its importing phase, before it links the objects into
|
||
|
|
the ObjectIndexTable. Most objects won't already exist, but some
|
||
|
|
may, and those that do must either be removed from their chunk, or
|
||
|
|
have their content byte-for-byte validated.
|
||
|
|
|
||
|
|
Removal from a chunk just means deleting it from the chunk's local
|
||
|
|
index, and not writing it to the global ObjectIndexTable. This
|
||
|
|
creates a hole in the chunk which is wasted space, and that isn't
|
||
|
|
very useful. Fortunately objects that fit fully within one chunk
|
||
|
|
may be easy to inflate and double check, as they are small. Objects
|
||
|
|
that are big span multiple chunks, and the new chunks can simply be
|
||
|
|
deleted from the ChunkTable, leaving the original chunks.
|
||
|
|
|
||
|
|
Deltas can be checked quickly by inflating the delta and checking
|
||
|
|
only the insertion point text, comparing that to the existing data
|
||
|
|
in the repository. Unfortunately the repository is likely to use a
|
||
|
|
different delta representation, which means at least one of them
|
||
|
|
will need to be fully inflated to check the delta against.
|
||
|
|
|
||
|
|
* DhtPackParser should handle small-huge-small-huge
|
||
|
|
|
||
|
|
Multiple chunks need to be open at once, in case we get a bad
|
||
|
|
pattern of small-object, huge-object, small-object, huge-object. In
|
||
|
|
this case the small-objects should be put together into the same
|
||
|
|
chunk, to prevent having too many tiny chunks. This is tricky to do
|
||
|
|
with OFS_DELTA. A long OFS_DELTA requires all prior chunks to be
|
||
|
|
closed out so we know their lengths.
|
||
|
|
|
||
|
|
* RepresentationSelector performance bad on Cassandra
|
||
|
|
|
||
|
|
The 1.8 million batch lookups done for linux-2.6 kills Cassandra, it
|
||
|
|
cannot handle this read load.
|
||
|
|
|
||
|
|
* READ_REPAIR isn't fully accurate
|
||
|
|
|
||
|
|
There are a lot of places where the generic DHT code should be
|
||
|
|
helping to validate the local replica is consistent, and where it is
|
||
|
|
not, help the underlying storage system to heal the local replica by
|
||
|
|
reading from a remote replica and putting it back to the local one.
|
||
|
|
Most of this should be handled in the DHT SPI layer, but the generic
|
||
|
|
DHT code should be giving better hints during get() method calls.
|
||
|
|
|
||
|
|
* LOCAL / WORLD writes
|
||
|
|
|
||
|
|
Many writes should be done locally first, before they replicate to
|
||
|
|
the other replicas, as they might be backed out on an abort.
|
||
|
|
|
||
|
|
Likewise some writes must take place across sufficient replicas to
|
||
|
|
ensure the write is not lost... and this may include ensuring that
|
||
|
|
earlier local-only writes have actually been committed to all
|
||
|
|
replicas. This committing to replicas might be happening in the
|
||
|
|
background automatically after the local write (e.g. Cassandra will
|
||
|
|
start to send writes made by one node to other nodes, but doesn't
|
||
|
|
promise they finish). But parts of the code may need to force this
|
||
|
|
replication to complete before the higher level git operation ends.
|
||
|
|
|
||
|
|
* Forks/alternates
|
||
|
|
|
||
|
|
Forking is common, but we should avoid duplicating content into the
|
||
|
|
fork if the base repository has it. This requires some sort of
|
||
|
|
change to the key structure so that chunks are owned by an object
|
||
|
|
pool, and the object pool owns the repositories that use it. GC
|
||
|
|
proceeds at the object pool level, rather than the repository level,
|
||
|
|
but might want to take some of the reference namespace into account
|
||
|
|
to avoid placing forked less-common content near primary content.
|