[prev in list] [next in list] [prev in thread] [next in thread]
List: helix-server-cvs
Subject: [Server-cvs] engine/dataflow fastfile.cpp,1.11,1.12
From: dcollins () helixcommunity ! org
Date: 2010-11-15 16:45:36
Message-ID: 201011151645.oAFGja2P029342 () mailer ! progressive-comp ! com
[Download RAW message or body]
Update of /cvsroot/server/engine/dataflow
In directory cvs01.internal.helixcommunity.org:/tmp/cvs-serv5841/server/engine/dataflow
Modified Files:
fastfile.cpp
Log Message:
Synopsis
========
Large-File Support updates for the Proxy Cache and CDist
Branches: HEAD
Suggested Reviewers: JJ (pending)
Description
===========
This contains updates related to Large-File Support in 64-bit server builds
for modules related to the Proxy Cache and CDist. For the most part,
32-bit builds should function the same as before. At over 13,000 lines
and half-a-megabyte, this diff is extremely huge!
This needs a lot more dev testing, especially with actual large files,
and I haven't actually tried CDist yet, but I was able to stream a
media file, saving it in the cache, and play it back from the cache.
This worked from both an updated server and from a v14 server.
The intention is that this will be both backward-compatable and 32/64-bit
cross compatable. A 32-bit proxy should cache up to 4GB of a larger file,
then go pass-through.
rncache:
~~~~~~~
Updated to read/write 64-bit integers to and from the cache. Generally,
most things were changed to 64-bits unless there was a reason why this
would be problematic. A bunch of 32-bit-specific integer manipulation was
moved into a new CacheInt class to simplify these changes. IHXValues4 is
used to communicate 64-bit quantities with CDist. (They're always set
as 64-bit values for simplicity, even on 32-bit builds.)
miiplin:
~~~~~~~
Countless updates across all classes. Updated to extract the new
64-bit stat data (described below.) Large-files are only supported
when retrieved via RTSP. The legacy binary MEI protocol was not updated
other than for compiler errors/warnings/etc.
meiffpln:
~~~~~~~~
Sets 64-bit stat values in the stat response (sent as an SDP message).
Since the old values were specifically named with "ulName" 32-bit integer
style names, we don't replace them but instead add new ones. This time,
the names chosen are more like "Size64", "ModificationTime64", etc.
The new values will be ignored by a 32-bit proxy or CDist subscriber,
which will continue to use the 32-bit values.
cdist:
~~~~~
Primarily updated to extract the 64-bit stat values passed to it from rncache.
siiplin:
~~~~~~~
Updated to build correctly with the new HX_OFF_T Seek() API, but does
not support 64-bit file sizes.
server/protocol/sdp:
~~~~~~~~~~~~~~~~~~~
Updated to enable extracting 64-bit integer quantities from an SDP message.
THe IHXSDPAttrib interface was deprecated, replacing with IHXSDPAttrib2.
common/include:
~~~~~~~~~~~~~~
Added the IHXSDPAttrib2 interface. Added a missing CLSID_IHXValues4
#define. Updated the IHXCacheFile, IHXCacheFileSetVersion, and
IHXFastFileStats interfaces for 64-bits (using HX_OFF_T, which remains
32-bits on 32-bit builds).
server/engine/dataflow/fastfile.cpp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RSS-related updates, and more 64-bit updates.
filesystem/local/full/smplfsys.cpp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Fixed a crash caused by an uninitialized character array pointer that was
found during dev testing.
server/engine/context/hxclfact.cpp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Updated to allow creation of IHXValues4 objects.
Files Affected
==============
server-restricted/cache/rncache/cacheint.h [new]
common/include/hxcache.h
common/include/hxfiles.h
common/include/hxpiids.h
common/include/ihxpckts.h
common/runtime/pub/hlxclib/stdlib.h
filesystem/local/full/smplfsys.cpp
server/engine/context/hxclfact.cpp
server/engine/core/_main.cpp
server/engine/dataflow/fastfile.cpp
server/engine/dataflow/pub/fastfile_factory.h
server/engine/dataflow/pub/fastfile.h
server/include/hxsdp.h
server/protocol/sdp/hxsdpp.cpp
server/protocol/sdp/hxsdpp.h
server/protocol/sdp/sdpstats.cpp
server-restricted/broadcast/transport/bcng/recv/strmsess.cpp
server-restricted/cache/meiffpln/meiffpln.cpp
server-restricted/cache/meiffpln/meiffpln.h
server-restricted/cache/meiplin/meikey.cpp
server-restricted/cache/meiplin/meikey.h
server-restricted/cache/miiplin/filesupp.cpp
server-restricted/cache/miiplin/filesupp.h
server-restricted/cache/miiplin/iids.cpp
server-restricted/cache/miiplin/meiconn.cpp
server-restricted/cache/miiplin/meiconn.h
server-restricted/cache/miiplin/meiresp.h
server-restricted/cache/miiplin/miientire.cpp
server-restricted/cache/miiplin/miientire.h
server-restricted/cache/miiplin/miifetch.cpp
server-restricted/cache/miiplin/miifetch.h
server-restricted/cache/miiplin/miifile.cpp
server-restricted/cache/miiplin/miifile.h
server-restricted/cache/miiplin/miifilep.cpp
server-restricted/cache/miiplin/miifilep.h
server-restricted/cache/miiplin/miilog.cpp
server-restricted/cache/miiplin/miilog.h
server-restricted/cache/miiplin/miiplin.cpp
server-restricted/cache/miiplin/miiplin.h
server-restricted/cache/miiplin/miipref.cpp
server-restricted/cache/miiplin/miipref.h
server-restricted/cache/miiplin/miistats.h
server-restricted/cache/miiplin/remoteconn.h
server-restricted/cache/miiplin/rmtcback.cpp
server-restricted/cache/miiplin/rmtcback.h
server-restricted/cache/miiplin/rmtfile.cpp
server-restricted/cache/miiplin/rmtfile.h
server-restricted/cache/miiplin/rtspconn.cpp
server-restricted/cache/miiplin/rtspconn.h
server-restricted/cache/miiplin/Umakefil
server-restricted/cache/rncache/cmegc.cpp
server-restricted/cache/rncache/cmegc.h
server-restricted/cache/rncache/datmgr.cpp
server-restricted/cache/rncache/datmgr.h
server-restricted/cache/rncache/fivemlist.cpp
server-restricted/cache/rncache/fivemlist.h
server-restricted/cache/rncache/fsmresp.cpp
server-restricted/cache/rncache/fsmresp.h
server-restricted/cache/rncache/fsresp.cpp
server-restricted/cache/rncache/fsresp.h
server-restricted/cache/rncache/rncache.cpp
server-restricted/cache/rncache/rncache.h
server-restricted/cache/rncache/rncadvise.cpp
server-restricted/cache/rncache/rncadvise.h
server-restricted/cache/rncache/rncf.cpp
server-restricted/cache/rncache/rncf.h
server-restricted/cache/rncache/Umakefil
server-restricted/cache/rncache/win32.pcf
server-restricted/cdist/cdadplin/cdadplin.cpp
server-restricted/cdist/cdadplin/cdadplin.h
server-restricted/cdist/cdadplin/cdadrule.cpp
server-restricted/cdist/cdadplin/cdadrule.h
server-restricted/cdist/cdistpln/cdistpln.cpp
server-restricted/cdist/cdistpln/cdistpln.h
server-restricted/proxy/proxylib/prxctxt.cpp
server-restricted/proxy/proxylib/prxroute.cpp
server-restricted/proxy/proxylib/prxymisc.cpp
server-restricted/proxy/proxylib/pub/prxroute.h
server-restricted/proxy/proxylib/pub/prxymisc.h
server_rn/cache/siiplin/mmsfob.cpp
server_rn/cache/siiplin/siiplin.cpp
Testing Performed
=================
Unit Tests:
- N/A
Integration Tests:
- Tested basic cold-cache and hot-cache playback with realvideo10.rm when playing
from a v14 server.
- Same from a modified HEAD server.
- Streaming a large rm file via the proxy was causing an assert, which
will be debugged post-checkin. The same file streams ok directly from
the origin.
- A ton more dev testing post-checkin when it's easier to build all the
platform/product variations.
Leak Tests:
- N/A
Performance Tests:
- N/A
Platforms Tested: linux-rhel5-x86_64
Builds Verified: linux-rhel5-x86_64
QA Hints
========
- N/A
Index: fastfile.cpp
===================================================================
RCS file: /cvsroot/server/engine/dataflow/fastfile.cpp,v
retrieving revision 1.11
retrieving revision 1.12
diff -u -d -r1.11 -r1.12
--- fastfile.cpp 2 Nov 2010 20:52:47 -0000 1.11
+++ fastfile.cpp 15 Nov 2010 16:45:33 -0000 1.12
@@ -115,8 +115,8 @@
#endif
// expressed as a percentage
-#define READ_RATIO 10
-#define READ_MIN 4 * 1024
+#define FF_READ_RATIO 10
+#define FF_READ_MIN (4 * 1024)
//////////////////////////////////////////////////////////////////////
@@ -128,7 +128,7 @@
, m_pContext(pContext)
, m_pMIIStats(NULL)
, m_bUseWrap2(0)
- , m_ulMaxMemUse(0)
+ , m_ulMaxMBUse(0)
{
m_pFastFileDict = new Dict;
@@ -149,9 +149,8 @@
INT32 temp = 0;
if (pRegistry)
- pRegistry->GetIntByName("config.FastFileMaxMemUse", temp);
-
- m_ulMaxMemUse = (UINT32)temp; //XXX not 64-bit safe
+ pRegistry->GetIntByName("config.FastFileMBMemUse", temp);
+ m_ulMaxMBUse = temp;
HX_RELEASE(pRegistry);
}
@@ -180,7 +179,7 @@
{
pWrapper = (IHXFileObject*)new FastFile(pFileObj, NULL,
m_pContext, ulBlockSize, this, m_pFastFileDict, m_pMIIStats,
- bAlignReads, bCacheStats, ulBlockSize, m_ulMaxMemUse);
+ bAlignReads, bCacheStats, ulBlockSize, m_ulMaxMBUse);
pWrapper->AddRef();
return HXR_OK;
@@ -202,7 +201,7 @@
{
pWrapper = (IHXFileObject*)new FastFile(pFileObj, NULL,
m_pContext, ulBlockSize, this, m_pFastFileDict, m_pMIIStats,
- bAlignReads, bCacheStats, ulMaxBlockSize, m_ulMaxMemUse);
+ bAlignReads, bCacheStats, ulMaxBlockSize, m_ulMaxMBUse);
pWrapper->AddRef();
return HXR_OK;
@@ -266,7 +265,7 @@
BOOL /*IN*/ bAlignReads,
BOOL /*IN*/ bCacheStats,
UINT32 /*IN*/ ulMaxBlockSize,
- UINT32 /*IN*/ ulMaxMemUse)
+ UINT32 /*IN*/ ulMaxMBUse)
: m_ulRefCount (0)
, m_pContext (pContext)
@@ -323,7 +322,7 @@
, m_bReadCancelled(0)
, m_ulBytesSinceLastRead(0)
, m_ulMaxBlockSize(ulMaxBlockSize)
- , m_ulMaxMemUse(ulMaxMemUse)
+ , m_ulMaxMBUse(ulMaxMBUse)
, m_pMemUse(0)
, m_pAlignmentReadBuf(0)
, m_ulChunkSize(0)
@@ -603,7 +602,7 @@
}
if (m_nRealReadPos + ulCount > m_nFileSize)
{
- m_ulRealReadSize = m_nFileSize - m_nRealReadPos;
+ m_ulRealReadSize = (UINT32)(m_nFileSize - m_nRealReadPos);
}
}
else
@@ -710,7 +709,7 @@
// if the time to use the data is less than the time to read then
// we need to increase our read size
- double target = READ_RATIO * msToUseData / 100;
+ double target = FF_READ_RATIO * msToUseData / 100;
if (m_ReadDuration >= target)
{
if (m_ulChunkSize < m_ulMaxBlockSize)
@@ -733,7 +732,7 @@
fflush(stdout);
#endif // VERBOSE_TIMING
}
- else if (m_ulChunkSize > READ_MIN)
+ else if (m_ulChunkSize > FF_READ_MIN)
{
// if we're aligning read the duration needs to be much less
// than the target before we'll adjust (since all we can do is
@@ -748,8 +747,8 @@
m_ulChunkSize = (UINT32)(m_ulChunkSize *
m_ReadDuration / target);
}
- if (m_ulChunkSize < READ_MIN)
- m_ulChunkSize = READ_MIN;
+ if (m_ulChunkSize < FF_READ_MIN)
+ m_ulChunkSize = FF_READ_MIN;
#ifdef VERBOSE_TIMING
printf("%p %f %f D%u\n",
this, msToUseData, m_ReadDuration, m_ulChunkSize);
@@ -759,10 +758,10 @@
}
BOOL bMaxHit = FALSE;
- if (m_ulMaxMemUse)
+ if (m_ulMaxMBUse)
{
// this isn't thread safe, but it doesn't need to be exact
- UINT32 ulLargestBlock = m_ulMaxMemUse - *m_pMemUse;
+ UINT32 ulLargestBlock = (UINT32)(((UINT64)m_ulMaxMBUse * 1024 * 1024) - *m_pMemUse);
if (ulLargestBlock < m_ulChunkSize)
{
if (m_bAlignReads)
@@ -816,17 +815,19 @@
// create a read size that will leave the file in an aligned
// position. First determine the end read position excluding alignment
- m_ulAlignmentReadAmount = m_nRealReadPos + m_ulRealReadSize;
+ HX_OFF_T nTemp = (UINT32)(m_nRealReadPos + m_ulRealReadSize);
// next align the position to the end of the block and subtract the
// start position to determine the amount that should be read
- m_ulAlignmentReadAmount = ((m_ulAlignmentReadAmount |
- (m_ulChunkSize - 1)) + 1) - (m_nStartReadPos);
+ m_ulAlignmentReadAmount =
+ (UINT32)(((nTemp | (m_ulChunkSize - 1)) + 1) - (m_nStartReadPos));
- xprintf(("m_nRealReadPos = %u, m_ulRealReadSize = %u, ulAmount = %u, "
- "finalPos = %u\n", m_nRealReadPos, m_ulRealReadSize,
- m_ulAlignmentReadAmount,
- m_nRealReadPos + m_ulAlignmentReadAmount));
+ xprintf(("m_nRealReadPos = %llu, m_ulRealReadSize = %lu, ulAmount = %lu, "
+ "finalPos = %llu\n",
+ (unsigned long long)m_nRealReadPos,
+ (unsigned long)m_ulRealReadSize,
+ (unsigned long)m_ulAlignmentReadAmount,
+ (unsigned long long)(m_nRealReadPos + m_ulAlignmentReadAmount)));
#ifdef ALLOW_SHORT_READS
if (pBuf)
@@ -1115,9 +1116,13 @@
*m_pAggInternalBytesRead += pBuffer->GetSize();
- UINT32 ulRequestedReadAmount = m_bAlignReads ?
- m_ulAlignmentReadAmount + m_nAlignmentReadOffset
- : m_ulChunkSize;
+ //XXXDC this can't be right, the read amount could be > 4GB!
+ //UINT32 ulRequestedReadAmount = m_bAlignReads ?
+ // m_ulAlignmentReadAmount + m_nAlignmentReadOffset
+ // : m_ulChunkSize;
+ // I think it's supposed to be more like (needs testing):
+ UINT32 ulRequestedReadAmount =
+ m_bAlignReads ? m_ulAlignmentReadAmount : m_ulChunkSize;
BOOL bShortRead = FALSE;
if (pBuffer->GetSize() != ulRequestedReadAmount)
@@ -1290,8 +1295,8 @@
{
const char* pURL = NULL;
pRequest->GetURL (pURL);
- int nLen = pURL ? strlen(pURL) : 0;
- int nOldLen = m_pURL ? strlen(m_pURL) : 0;
+ int nLen = pURL ? (int)strlen(pURL) : 0;
+ int nOldLen = m_pURL ? (int)strlen(m_pURL) : 0;
BOOL bIgnore = FALSE;
xprintf(("%p: FastFile::SetRequest url=%s old url %s\n",
@@ -1924,7 +1929,7 @@
return 0;
}
- UINT32 ulNewCount = ulBlockSize - (nOffset - nBlockOffset);
+ UINT32 ulNewCount = ulBlockSize - (UINT32)(nOffset - nBlockOffset);
if (ulNewCount == 0 || ulNewCount > ulCount)
{
return 0;
_______________________________________________
Server-cvs mailing list
Server-cvs@helixcommunity.org
http://lists.helixcommunity.org/mailman/listinfo/server-cvs
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic