[prev in list] [next in list] [prev in thread] [next in thread] 

List:       linux-xfs
Subject:    Re: [Myricom help #36078] application sleeps indefinitely trying
From:       Myricom Technical Support <help () myri ! com>
Date:       2005-08-30 22:53:40
Message-ID: 4314E374.4060002 () myri ! com
[Download RAW message or body]

Ashutosh,

 From one of our GM software developers:

I suggest they try the attached patch which should solve their problem.

Basically the incompatibility is that gm-registration of any
"process-private" memory requires the gm driver to lock the
corresponding pages for the duration of the registration (which is the
life of the gm-port for gm_dma_malloc() buffers). During that time
those pages cannot be used for reading through O_DIRECT (because
O_DIRECT also requires obtaining a lock on the pages at some point).

The patch changes gm_dma_malloc() to allocate memory with
map(MAP_SHARED|MAP_ANONYMOUS). The gm-registration of such pages does
not require a lock and should be compatible with any O_DIRECT activity
(or any other activity that we can think of at this point).

Note that this patch potentially introduces a small semantic change
for programs that uses fork(), but should not be a problem as we don't
expect programs using forks() to rely on any specific sharing
semantics for the gm_dma_malloc'ed() buffers.

Please apply this patch to GM-2 and let us know how this works for you.

Thanks,
Susan


-- 
Susan Blackford
Member of Technical Staff
Myricom Inc.

["dma-odirect.patch" (text/x-patch)]

Index: libgm/gm_dma_malloc.c
===================================================================
RCS file: /repository/gm/libgm/gm_dma_malloc.c,v
retrieving revision 1.8.22.1
diff -u -r1.8.22.1 gm_dma_malloc.c
--- libgm/gm_dma_malloc.c	28 May 2004 19:09:10 -0000	1.8.22.1
+++ libgm/gm_dma_malloc.c	30 Aug 2005 20:55:55 -0000
@@ -246,6 +246,26 @@
 }
 #endif /* GM_KERNEL */
 
+
+#if GM_KERNEL || !GM_OS_LINUX
+#define gm_alloc_pages_for_dma gm_alloc_pages
+#define gm_alloc_free_pages_for_dma gm_free_pages
+#else
+#include <sys/mman.h>
+
+void *gm_alloc_pages_for_dma(gm_size_t length)
+{
+  void * ptr = mmap(0, length, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+  return (ptr == MAP_FAILED) ? 0 : ptr;
+}
+
+void gm_free_pages_for_dma(void *addr, gm_size_t length)
+{
+  munmap(addr, length);
+}
+#endif
+
+
 /****************************************************************
  * Environment abstraction
  *
@@ -272,7 +292,7 @@
   
     /* allocate some pages */
     
-    addr = gm_alloc_pages (length);
+    addr = gm_alloc_pages_for_dma (length);
     if (addr == 0)
       {
 	status = GM_OUT_OF_MEMORY;
@@ -300,7 +320,7 @@
     GM_RETURN_STATUS (GM_SUCCESS);
     
   abort_with_pages:
-    gm_free_pages (addr, length);
+    gm_free_pages_for_dma (addr, length);
   abort_with_nothing:
     GM_RETURN_STATUS (status);
   }
@@ -447,7 +467,7 @@
 #if !GM_KERNEL && GM_CAN_REGISTER_MEMORY
   {
     gm_deregister_memory (p, ptr, length);
-    gm_free_pages (ptr, length);
+    gm_free_pages_for_dma (ptr, length);
   }
 #elif !GM_KERNEL && !GM_CAN_REGISTER_MEMORY
   {



[prev in list] [next in list] [prev in thread] [next in thread]