diff --git a/doc/DEV.md b/doc/DEV.md index 689d2332..8d21ad5e 100644 --- a/doc/DEV.md +++ b/doc/DEV.md @@ -57,8 +57,7 @@ Handling issues Issues are opened in the bug tracker of the project (typically https://github.com/P-p-H-d/mlib/issues ) -If the issue cannot be closed and taken into account in less than (typically) one month, -or before a release, +If the issue cannot be taken into account quickly and are long term, it shall be moved into the long term issues included in the repository: https://github.com/P-p-H-d/mlib/blob/master/doc/ISSUES.org This is a file in ORG mode: the issues are created with the proper tags. diff --git a/doc/ISSUES.org b/doc/ISSUES.org index 8cfbe94e..6e1e8c01 100644 --- a/doc/ISSUES.org +++ b/doc/ISSUES.org @@ -5,17 +5,34 @@ Specialize Hashmap for huge objects on supported hardware: -based on OA hash map with taggued pointer (needs space for tagging): -low bits = low bits of hash +Based on OA hash map with taggued pointer (needs space for tagging): +low bits = high bits of hash pointer to object { key, hash, value } -Can detect collision without dereferencing object with 1/2^lowbits chances. +Can solve collision without dereferencing object with 1/2^lowbits chances. +NB: hash may be uneeded due to the small chances of collision. Reduce cache usage ==> Should be faster. +Increase cache usage for very small object (integer). Use of pointer to avoid moving object on GC. On AMD64, only 48 bits of 64 bits are really used for addressing data. +==> lowbits= 16 +Empty representation= 0 (hash can be 0. pointer cannot be it). +Deleted representation = 1 (pointer cannot be zero) +<---- PTR ----><-HASH-> + +#+BEGIN_SRC C +#define LOW_BITS 16 +#define ALIGNED_BITS 2 +void *get_ptr(int64_t x) { + return (void*) ((x >> LOW_BITS) << ALIGNED_BITS); +} +unsigned get_small_hash(int64_t x) { + return x & ((1< Analyse limitation and possible constraint usages. -* TODO #20: Bucket priority queue :ENHANCEMENT: +* TODO #20: New: Bucket priority queue :ENHANCEMENT: Add a new kind of priority queue. See https://en.wikipedia.org/wiki/Bucket_queue @@ -145,7 +162,7 @@ Check if we can use BITSET, or introduce fixed size BITSET or use ad-hock implementation. -* TODO #19: Intrusive Red Black Tree :ENHANCEMENT: +* TODO #19: New: Intrusive Red Black Tree :ENHANCEMENT: Add intrusive red black tree. Look also for AVL tree (NOTE: Is there a performance difference between the two?) @@ -158,9 +175,9 @@ See the cells in yellow here: http://htmlpreview.github.io/?https://github.com/P-p-H-d/mlib/blob/master/doc/Container.html -* TODO #17: Ressource handler :ENHANCEMENT: +* TODO #17: New: Ressource handler :ENHANCEMENT: - A ressouce handler which shall associated a unique handle to a ressource. + A global 'ressouce handler' which shall associated a unique handle to a ressource. The handle shall be unique for the ressource and shall not be reused. It is typically a 64 bits integers always incremented (even if the program creates one billion ressources per second, the counter won't overflow @@ -189,7 +206,7 @@ http://htmlpreview.github.io/?https://github.com/P-p-H-d/mlib/blob/master/doc/Co * embedded the type in the ressource handler: more work, API more complex. Memory usage seems better. -* TODO #16: Lock Free List :ENHANCEMENT: +* TODO #16: New: Lock Free List :ENHANCEMENT: Implement a lock free list. Most of the difficulty is the memory reclamation part. Typically this lock free list shall be compatible with RCU method. @@ -271,7 +288,7 @@ Needs some formal operator in the oplist to support it fully and an example. It is a kind of object inheritance where the container inherits some extra data from its base. -* TODO #12: Atomic shared pointer :ENHANCEMENT: +* TODO #12: New: Atomic shared pointer :ENHANCEMENT: Add an extension to the SHARED_PTR API: @@ -347,11 +364,14 @@ Other implementation seems to have it hard to be lock-free: cf. https://github.c -* TODO #5: Concurrent dictionary Container :ENHANCEMENT: +* TODO #5: New: Concurrent dictionary Container :ENHANCEMENT: Implement a more efficient dictionary than lock + std dictionary for all operations when dealing with threads. See https://msdn.microsoft.com/en-us/library/dd287191(v=vs.110).aspx +The best parallel algorithm is still when there is as few synchronization as possible. A concurrent dictionary will fail at this and will result in average performance at best. +The typical best case will be in RCU context (a lot of readers, few writers), so the interface shall be compatible with such structure. + ** Multiple locks within the dictionnary A potential implementation may be to request at initialization time the number of concurrent thread N. @@ -372,33 +392,25 @@ See also https://github.com/simonhf/sharedhashfile ** Lock Free dictionnary Evaluate also lock-free dictionary (easier with open addressing). -It needs a complete rewrite of the inner loop through. The hard part is the dynamic resizing of the internal array (see http://preshing.com/20160222/a-resizable-concurrent-map/ for a potential solution and http://www.cs.toronto.edu/~tomhart/papers/tomhart_thesis.pdf for memory reclamation techniques). See also https://www.research.ibm.com/people/m/michael/spaa-2002.pdf -https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.2017.11.22a.pdf persents different techniques used by linux kernel. +It needs a complete rewrite of the inner loop through. The hard part is the dynamic resizing of the internal array. +Ssee http://preshing.com/20160222/a-resizable-concurrent-map/ for a potential solution + and http://www.cs.toronto.edu/~tomhart/papers/tomhart_thesis.pdf for memory reclamation techniques). + and https://www.research.ibm.com/people/m/michael/spaa-2002.pdf +https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.2017.11.22a.pdf presents different techniques used by linux kernel. It needs before lock-free list: http://www.cse.yorku.ca/~ruppert/papers/lfll.pdf http://www.cse.yorku.ca/~ruppert/Mikhail.pdf -** Context - -The best parallel algorithm is still when there is as few synchronization as possible. A concurrent dictionary will fail at this and will result in average performance at best. -The typical best case will be in RCU context (a lot of readers, few writers), so the interface shall be compatible with such structure. - -** Linked list - -Another solution is to create a huge list of items which is: - -- atomically updated, -- in reverse order of the hash (bit 0 is highest bit, Bit 63 is the lowest bit). - -Hash table will only give quick access to items to this list. Expanding the table won't change the order of the list (so the items will remain at the same place and always accessible by other threads) but just add sentinel in the sentinel at the right place (there won't be any reallocation of the list). Inserting will need to insert an item at the right place. -Not sure it will really help. To analyse. - - Concurrent dictionary is possible now with CONCURRENT_DEF + DICT_DEF. - But it uses a global lock for all dictionary access. - -** Open Addressing Indirect Table with a lock free node pool used as transaction pool. - A good way maybe Open Addessing table used only for indirection and a freelist memory reclamation container for handle the entries (like a transaction). + Solution? + -------- + + Use of the concurrent pool (m-c-mempool) to allocate items. + The big table will only store pointers (+ part of hash in unused bits?) to such allocate items. + Atomic update is done by allocating a new node, update it, and atomicaly store it in the dict, putting the old one as logically deleted and to reclaim later. + (Other threads may still read its data so we cannot free immediatly). + + * DONE #22: Enhanced services for SPSC Queue :ENHANCEMENT: Add services: