[prev in list] [next in list] [prev in thread] [next in thread]
List: opensolaris-driver-discuss
Subject: Re: [driver-discuss] ddi_dma_attr_t, dma_attr_maxxfer quesiton.
From: Carlos Cumming <tcc_sun29 () thinkthink ! com>
Date: 2007-12-05 17:33:34
Message-ID: 4756E0EE.1010101 () thinkthink ! com
[Download RAW message or body]
Ok! Now I'm getting the picture. I'm revisiting it right now.
Thanks!
Carlos
Garrett D'Amore wrote:
> Carlos Cumming wrote:
>> I don't follow how your example is apropos...
>>
>> I think I found the answer myself by looking at
>> scsi_hba_attach_setup(.. ) in .../common/io/scsi/impl/scsi_hba.c.
>> From how I read the various documentation and examples I expected
>> scsi_hba_attach_setup(.. ) to squirrel away dma_attr_maxxfer, then
>> when requests come down they would be chunked into dma_attr_maxxfer
>> chunks. ...NOT!
>>
>> It seems that scsi_hba_attach_setup(.. ) only saves
>> dma_attr_/minxfer/ and dma_attr_burstsizes. Since it didn't save away
>> dma_attr_/maxxfer/ there's no information to limit requests coming
>> down until xxx_tran_init_pkt(.. ) gets with the full request size,
>> and I respond with a partial allocation.
>>
>> I was hoping to not have to implement all of the partial allocation
>> code by limiting the size of the requests even before they got to me.
>> It seemed (to me) to be something that should be handled by my caller
>> and I was stubbornly trying to get it to do it.
>>
>
> What I'm saying is that you don't have to do that. There is logic in
> the SCSI framework to do it. But you supply tran_setup_pkt and
> tran_teardown_pkt instead. Then the framework supplies a default
> tran_init_pkt() and tran_destroy_pkt() that does all that work of
> using partial mappings for you. The end result being that
> tran_start() only has to deal with the cookies that are supplied in
> the pkt. (The DMA setup and binding is already done for you, using
> the DMA attributes that were pushed in via scsi_hba_setup().) You can
> even have a scatter-gather list length of one, and the framework will
> avoid passing you down more than one cookie at a time. (Which is nice
> for simplicity, even if it isn't maximally performant.)
>
> Again, look at blk2scsa.c to see how it is done. Unfortunately,
> tran_start() in blk2scsa doesn't tell the whole story, because its
> just a translation layer. But b2s_request_dma() in blk2scsa.c returns
> the DMA cookies to the driver being translated for, and the driver can
> just use them.
>
> Some of what I'm talking about isn't properly documented (there are
> stale man pages and such), but the code at least is correct. And the
> API is public, it just needs to be better documented.
>
> -- Garrett
>
>> Thanx, Carlos
>>
>> Garrett D'Amore wrote:
>>
>>> Carlos Cumming wrote:
>>>
>>>> Mark Johnson wrote:
>>>>
>>>>
>>>>> You should support partial mappings. You can see 1M transfers
>>>>> come down the scsi stack. You would
>>>>> want that broken into multiple 128K DMA "windows".
>>>>>
>>>> My understanding was that the call to scsi_hba_attach_setup(dev,
>>>> &hba_dma_attr, softs->tran, 0) would limit requests I see to the
>>>> attributes defined by hba_dma_attr. It might even be more efficient
>>>> for my caller to be handling it than the hba driver.
>>>>
>>> Yes. There is nice and efficient support for DMA partial mappings
>>> in the SCSA framework. To do this you might want to look at
>>> tran_setup_pkt and tran_destroy_pkt. Then the SCSA framework will
>>> give you DMA chunks that are already broken up (actually using
>>> partial mappings) by stashing DMA cookies in the SCSI pkt as
>>> pkt->pkt_cookies and pkt->pkt_ncookie or somesuch.
>>>
>>> Have a look at my blk2scsa layer generic layer to see an example.
>>> http://cr.opensolaris.org/~gdamore/blk2scsa/
>>>
>>> (Note that blk2scsa is a bit more complicated because it also
>>> supports "generic" layers that might not even support DMA. But it
>>> assumes DMA by default and is optimized for the DMA case.)
>>>
>>> -- Garrett
>>>
>>>
>>>> Thanks, Carlos
>>>>
>>>>
>>>>> e.g. I believe /dev/rdsk/c* is case where this
>>>>> will happen.
>>>>>
>>>>>
>>>>>
>>>>>>> e.g. if a dma engine has a 16 bit size for the DMA engine, the
>>>>>>> max size it can DMA is 64K - 1 byte. (0xffff). A few DMA
>>>>>>> engines will use 0 to signify the 0x10000 value, but that means
>>>>>>> adding another bit to the counter or adding some more
>>>>>>> complicated logic.
>>>>>>>
>>>>>> [the above] is how I understood it.... But if you want to get
>>>>>> optimal performance out of your hw, then you'd want to do a full
>>>>>> 64k DMA (somehow). Xfering 65535 (64k - 1) bytes is a really
>>>>>> oddball value. Specially if you have a block (512 byte) device
>>>>>> like I have.
>>>>>>
>>>>>> In my case I'm actually, "logically" limited. I.e., the firmware
>>>>>> in the device is limiting me to 128k, the actual counter goes to
>>>>>> 2^32 - 1 (to use your parlance). So I take it in my case using a
>>>>>> full value of 128k (i.e., 0x20000) in dma_attr_maxxfer is correct.
>>>>>>
>>>>> yes.
>>>>>
>>>>>
>>>>>
>>>>> MRJ
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Thanks!!!
>>>>>>
>>>>>> Carlos
>>>>>>
>>>>>>> Carlos Cumming wrote:
>>>>>>>
>>>>>>>> I'm writing a SCSI driver. The board I'm talking to can do a
>>>>>>>> max 128k (256 blocks) DMA. I am using the following two
>>>>>>>> dma_attr_t's:
>>>>>>>>
>>>>>>>> static ddi_dma_attr_t hba_dma_attr = {
>>>>>>>> DMA_ATTR_V0, // dma_attr_version
>>>>>>>> Version of this structure
>>>>>>>> (u64)0, // dma_attr_addr_lo
>>>>>>>> Lowest usable address
>>>>>>>> (u64)0xffffffffffffffff, // dma_attr_addr_hi
>>>>>>>> Highest usable address
>>>>>>>> (u64)(256 * 512) - 1, // dma_attr_count_max
>>>>>>>> Maximum DMAable byte count (1 sge * maxxfer)
>>>>>>>> 512, // dma_attr_align
>>>>>>>> Minimum alignment (This value correct?)
>>>>>>>> 0x7f, // dma_attr_burstsizes
>>>>>>>> Burst sizes (??? big values don't work)
>>>>>>>> (u64)4, // dma_attr_minxfer
>>>>>>>> Minimum transfer
>>>>>>>> (u64)(256 * 512) - 1, // dma_attr_maxxfer
>>>>>>>> Maximum transfer count for entire I/O.
>>>>>>>> (u64)0xffffffffffffffff, // dma_attr_seg
>>>>>>>> Maximum segment (boundry DMA engine cannot cross).
>>>>>>>> 512, // dma_attr_sgllen
>>>>>>>> Maximum scatter gather list entries
>>>>>>>> 512, // dma_attr_granular
>>>>>>>> Granularity
>>>>>>>> 0, // dma_attr_flags
>>>>>>>> Flags (reserved)
>>>>>>>> };
>>>>>>>>
>>>>>>>> static ddi_dma_attr_t cmd_dma_attr = {
>>>>>>>> DMA_ATTR_V0, // dma_attr_version
>>>>>>>> Version of this structure
>>>>>>>> (u64)0, // dma_attr_addr_lo
>>>>>>>> Lowest usable address
>>>>>>>> (u64)0xffffffffffffffff, // dma_attr_addr_hi
>>>>>>>> Highest usable address
>>>>>>>> (u64)(256 * 512) - 1, // dma_attr_count_max
>>>>>>>> Maximum DMAable byte count (1 sge * maxxfer)
>>>>>>>> 512, // dma_attr_align
>>>>>>>> Minimum alignment (This value correct?)
>>>>>>>> 0x7f, // dma_attr_burstsizes
>>>>>>>> Burst sizes (??? big values don't work)
>>>>>>>> (u64)512, // dma_attr_minxfer
>>>>>>>> Minimum transfer
>>>>>>>> (u64)(256 * 512) - 1, // dma_attr_maxxfer
>>>>>>>> Maximum transfer count for entire I/O.
>>>>>>>> (u64)0xffffffffffffffff, // dma_attr_seg
>>>>>>>> Maximum segment (boundry DMA engine cannot cross).
>>>>>>>> 512, // dma_attr_sgllen
>>>>>>>> Maximum scatter gather list entries
>>>>>>>> 512, // dma_attr_granular
>>>>>>>> Granularity
>>>>>>>> 0, // dma_attr_flags
>>>>>>>> Flags (reserved)
>>>>>>>> };
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Some time later, I use the first one:
>>>>>>>>
>>>>>>>> if (scsi_hba_attach_setup(dev, &hba_dma_attr, softs->tran, 0)
>>>>>>>> != DDI_SUCCESS) ..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Then even later (but still within attach if that matters) I use
>>>>>>>> cmd_dma_attr...
>>>>>>>>
>>>>>>>> if (ddi_dma_alloc_handle(softs->dev, &cmd_dma_attr,
>>>>>>>> DDI_DMA_SLEEP, NULL, &softs->dma_hndl) != DDI_SUCCESS) {
>>>>>>>> cmn_err(CE_WARN, "tw%d: tw_create_reqs: Cannot alloc
>>>>>>>> dma handle for command blocks.", inst);
>>>>>>>> return(1);
>>>>>>>> }
>>>>>>>>
>>>>>>>> #define MEMSIZE (256 * 512)
>>>>>>>>
>>>>>>>> if (ddi_dma_mem_alloc(softs->dma_hndl, MEMSIZE, &accattr,
>>>>>>>> MEMTYPE, DDI_DMA_SLEEP, NULL, &mem, &rlen, &softs->acc_hndl)
>>>>>>>> != DDI_SUCCESS) {
>>>>>>>> cmn_err(CE_WARN, "tw%d: tw_create_reqs: Cannot alloc
>>>>>>>> dma memory for command blocks.", inst);
>>>>>>>> goto free_hndl;
>>>>>>>> }
>>>>>>>>
>>>>>>>> {
>>>>>>>> int a;
>>>>>>>>
>>>>>>>> if ((a = ddi_dma_addr_bind_handle(softs->dma_hndl, NULL,
>>>>>>>> mem, rlen, MEMTYPE, DDI_DMA_SLEEP, NULL, &dmacookie,
>>>>>>>> &num_dmacookies)) != DDI_DMA_MAPPED) {
>>>>>>>> cmn_err(CE_WARN, "tw%d: tw_create_reqs: Cannot bind dma
>>>>>>>> memory to handle for command blocks. %d", inst, a);
>>>>>>>> goto free_mem;
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> The above ddi_dma_addr_bind_handle(.. ) _FAILS. It prints
>>>>>>>> "Cannot bind dma memory to handle for command blocks. 1". The
>>>>>>>> trailing ,"1" is DDI_ENOMEM if I understand it correctly.
>>>>>>>>
>>>>>>> What is MEMTYPE set to. It looks like you have set
>>>>>>> DDI_DMA_PARTIAL which is correct.
>>>>>>>
>>>>>>> You are not getting a failure returned. You are getting
>>>>>>> DDI_DMA_PARTIAL_MAP which means you have multiple DMA
>>>>>>> windows.
>>>>>>>
>>>>>>> #define DDI_DMA_PARTIAL_MAP 1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Now, here's where I get lost. If you look at
>>>>>>>> cmd_dma_attr.dma_attr_maxxfer above, it's set to (256 * 512) -
>>>>>>>> 1. I'm trying to get 256 * 512 of memory. So, it makes sense
>>>>>>>> that it fails ENOMEM, because I'm requesting one more byte than
>>>>>>>> dma_attr_maxxfer.
>>>>>>>>
>>>>>>>> Or does it make sense...
>>>>>>>>
>>>>>>>> _Every_ example that I look at shows dma_attr_maxxfer set to a
>>>>>>>> power of 2 - 1. I.e., the way everyone else uses
>>>>>>>> dma_attr_maxxfer is as if it's the max value of a dma counter,
>>>>>>>> just like dma_attr_count_max.
>>>>>>>>
>>>>>>>> So I'm feeling just a bit squeamish using a nice even power of
>>>>>>>> two value for dma_attr_maxxfer when nobody else is using it
>>>>>>>> that way. I figure they know something I don't.
>>>>>>>>
>>>>>>>> Can someone give me a clue???
>>>>>>>>
>>>>>>> The reason folks use power of 2's - 1 for things like maxxfer, seg,
>>>>>>> dma count is because counters are usually powers of two :-)
>>>>>>>
>>>>>>> e.g. if a dma engine has a 16 bit size for the DMA engine,
>>>>>>> the max size it can DMA is 64K - 1 byte. (0xffff). A few DMA
>>>>>>> engines will use 0 to signify the 0x10000 value, but that
>>>>>>> means adding another bit to the counter or adding some more
>>>>>>> complicated logic.
>>>>>>>
>>>>>>>
>>>>>>> MRJ
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> driver-discuss mailing list
>>>>>> driver-discuss@opensolaris.org
>>>>>> http://mail.opensolaris.org/mailman/listinfo/driver-discuss
>>>>>>
>>>> _______________________________________________
>>>> driver-discuss mailing list
>>>> driver-discuss@opensolaris.org
>>>> http://mail.opensolaris.org/mailman/listinfo/driver-discuss
>>>>
>>>>
>>
>> _______________________________________________
>> driver-discuss mailing list
>> driver-discuss@opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/driver-discuss
>>
>>
>
_______________________________________________
driver-discuss mailing list
driver-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic