From llcbench at cs.utk.edu  Wed May 21 20:23:07 2008
From: llcbench at cs.utk.edu (llcbench at cs.utk.edu)
Date: Wed, 21 May 2008 18:23:07 -0600
Subject: [Llcbench] Bug in Cachebench's data reporting
Message-ID: <4834BCEB.1070303@lanl.gov>

There's a bug in most of Cachebench's benchmark functions that causes the
reported bandwidth to appear to *increase* at large vector sizes.  Here's
a typical instance of the problem (from benchmark_cache_wonly()):

   alarm(duration);
   TIMER_START;

   while (keepgoing)
     {
       for  (index = 0; index < limit; index++)
	x[index] = wval;
       loops++;
     }

   TIMER_STOP;

            .
            .
            .

   double refcnt = ((double)loops*(double)limit)+(double)index;

The code specifies that keepgoing should be set to zero after duration seconds.
Then, while keepgoing is nonzero, limit array-touching iterations
(constituting one complete loop) are performed.  At the end of the loop,
the clock is stopped and the number of array-touching iterations is
computed as the number of complete loops times the number of iterations per
loop plus the leftover iterations from any incomplete loops.

The problem is that there are *never* any incomplete loops because keepgoing
is checked only at loop boundaries.  Hence, one extra complete set of
iterations is attributed to refcnt, making the benchmark appear to have
performed an artificially large amount of work per unit time.  At large
vector sizes, limit may be fairly large relative to loops, which can
significantly affect the reported performance.

The solution is to drop the "+(double)index" everywhere the inner loop can
never be interrupted.  I've attached a patch file that I believe corrects
the bug everywhere it appears.

Regards,
-- Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cachebench-refcnt.patch
Type: text/x-patch
Size: 2029 bytes
Desc: not available
Url : http://lists.cs.utk.edu/pipermail/llcbench/attachments/20080521/cd555a75/cachebench-refcnt.bin 

From llcbench at cs.utk.edu  Fri May 23 08:47:49 2008
From: llcbench at cs.utk.edu (llcbench at cs.utk.edu)
Date: Fri, 23 May 2008 14:47:49 +0200
Subject: [Llcbench] Bug in Cachebench's data reporting
In-Reply-To: <4834BCEB.1070303@lanl.gov>
References: <4834BCEB.1070303@lanl.gov>
Message-ID: <5EE5414F-160F-4DDB-AE44-409529729AF8@cs.utk.edu>

Thanks for the patch. I will make sure it makes it into CVS.

Phil

On May 22, 2008, at 2:23 AM, llcbench at cs.utk.edu wrote:

> There's a bug in most of Cachebench's benchmark functions that  
> causes the
> reported bandwidth to appear to *increase* at large vector sizes.   
> Here's
> a typical instance of the problem (from benchmark_cache_wonly()):
>
>  alarm(duration);
>  TIMER_START;
>
>  while (keepgoing)
>    {
>      for  (index = 0; index < limit; index++)
> 	x[index] = wval;
>      loops++;
>    }
>
>  TIMER_STOP;
>
>           .
>           .
>           .
>
>  double refcnt = ((double)loops*(double)limit)+(double)index;
>
> The code specifies that keepgoing should be set to zero after  
> duration seconds.
> Then, while keepgoing is nonzero, limit array-touching iterations
> (constituting one complete loop) are performed.  At the end of the  
> loop,
> the clock is stopped and the number of array-touching iterations is
> computed as the number of complete loops times the number of  
> iterations per
> loop plus the leftover iterations from any incomplete loops.
>
> The problem is that there are *never* any incomplete loops because  
> keepgoing
> is checked only at loop boundaries.  Hence, one extra complete set of
> iterations is attributed to refcnt, making the benchmark appear to  
> have
> performed an artificially large amount of work per unit time.  At  
> large
> vector sizes, limit may be fairly large relative to loops, which can
> significantly affect the reported performance.
>
> The solution is to drop the "+(double)index" everywhere the inner  
> loop can
> never be interrupted.  I've attached a patch file that I believe  
> corrects
> the bug everywhere it appears.
>
> Regards,
> -- Scott
> --- cachebench/cachebench.c.ORIG	2008-05-21 18:02:11.000000000 -0600
> +++ cachebench/cachebench.c	2008-05-21 18:04:11.000000000 -0600
> @@ -428,7 +428,7 @@
>   *oloops = loops;
>   *ous = TIMER_ELAPSED;
>   {
> -    double refcnt = ((double)loops*(double)limit)+(double)index;
> +    double refcnt = ((double)loops*(double)limit);
>     DBG(fprintf(stderr,"T: %ld loops at limit %ld took %f us, %f  
> refs.\n",loops,limit,*ous,refcnt));
>     return(refcnt);
>   }
> @@ -470,7 +470,7 @@
>   *oloops = loops;
>   *ous = TIMER_ELAPSED;
>   {
> -    double refcnt = ((double)loops*(double)limit)+(double)index;
> +    double refcnt = ((double)loops*(double)limit);
>     DBG(fprintf(stderr,"T: %ld loops at limit %ld took %f us, %f  
> refs.\n",loops,limit,*ous,refcnt));
>     return(refcnt);
>   }
> @@ -503,7 +503,7 @@
>   *oloops = loops;
>   *ous = TIMER_ELAPSED;
>   {
> -    double refcnt = ((double)loops*(double)limit)+(double)index;
> +    double refcnt = ((double)loops*(double)limit);
>     DBG(fprintf(stderr,"T: %ld loops at limit %ld took %f us, %f  
> refs.\n",loops,limit,*ous,refcnt));
>     return(refcnt);
>   }
> @@ -548,7 +548,7 @@
>   *oloops = loops;
>   *ous = TIMER_ELAPSED;
>   {
> -    double refcnt = ((double)loops*(double)limit)+(double)index;
> +    double refcnt = ((double)loops*(double)limit);
>     DBG(fprintf(stderr,"T: %ld loops at limit %ld took %f us, %f  
> refs.\n",loops,limit,*ous,refcnt));
>     return(refcnt);
>   }
> @@ -579,7 +579,7 @@
>   *oloops = loops;
>   *ous = TIMER_ELAPSED;
>   {
> -    double refcnt = ((double)loops*(double)limit)+(double)index;
> +    double refcnt = ((double)loops*(double)limit);
>     DBG(fprintf(stderr,"T: %ld loops at limit %ld took %f us, %f  
> refs.\n",loops,limit,*ous,refcnt));
>     return(refcnt);
>   }
> @@ -621,7 +621,7 @@
>   *oloops = loops;
>   *ous = TIMER_ELAPSED;
>   {
> -    double refcnt = ((double)loops*(double)limit)+(double)index;
> +    double refcnt = ((double)loops*(double)limit);
>     DBG(fprintf(stderr,"T: %ld loops at limit %ld took %f us, %f  
> refs.\n",loops,limit,*ous,refcnt));
>     return(refcnt);
>   }
> _______________________________________________
> Llcbench mailing list
> Llcbench at cs.utk.edu
> http://lists.cs.utk.edu/listinfo/llcbench


From llcbench at cs.utk.edu  Fri May 23 08:52:47 2008
From: llcbench at cs.utk.edu (llcbench at cs.utk.edu)
Date: Fri, 23 May 2008 08:52:47 -0400 (EDT)
Subject: [Llcbench] CVS-PAPI
Message-ID: <20080523125247.7B71F303C3@earth.cs.utk.edu>


Local user: uid=396(mucci) gid=731(dongarra) groups=731(dongarra),753(llcbench),2023(papi),2185(ti06)
CVS user: mucci

llcbench CHANGES

Fri May 23 08:52:47 EDT 2008

Update of /cvs/homes/llcbench/llcbench
In directory earth:/home/mucci/work/llcbench

Modified Files:
	CHANGES 
Log Message:
Patch from Scott for computing array indices

From llcbench at cs.utk.edu  Fri May 23 08:52:47 2008
From: llcbench at cs.utk.edu (llcbench at cs.utk.edu)
Date: Fri, 23 May 2008 08:52:47 -0400 (EDT)
Subject: [Llcbench] CVS-PAPI
Message-ID: <20080523125247.8FAB6303B6@earth.cs.utk.edu>


Local user: uid=396(mucci) gid=731(dongarra) groups=731(dongarra),753(llcbench),2023(papi),2185(ti06)
CVS user: mucci

llcbench/cachebench cachebench.c

Fri May 23 08:52:47 EDT 2008

Update of /cvs/homes/llcbench/llcbench/cachebench
In directory earth:/home/mucci/work/llcbench/cachebench

Modified Files:
	cachebench.c 
Log Message:
Patch from Scott for computing array indices