Using PAPI in PETSc code

Question

I am trying to count the number of cache misses, total cycles, etc. per iteration of a for loop inside MatMult_SeqSBAIJ_2() . I've configured PETSc to use PAPI and I am using the PAPI API like shown in the skeletal code below. PAPI always fails (!PAPI_OK) while running ./main . Is there a setting in PAPI or PETSc that I am missing, which enables proper counter sampling inside PETSc?

main.c:

PetscInitialize(&argc,&args,(char*)0,help);
PAPI_library_init(PAPI_VER_CURRENT); //Initialize PAPI
/*Matrix and vector setup code here. */
MatMult_SeqSBAIJ_2(A,xx,zz)
/*Destroy matrix here*/

sbaij2.c :

PetscErrorCode MatMult_SeqSBAIJ_2(Mat A,Vec xx,Vec zz)
{
long long counters[2];
int PAPI_events[2] = {PAPI_TOT_CYC, PAPI_L1_MISSES};
int retval=0;   

PAPI_start_counters( PAPI_events, 2 );

for (i=0; i<mbs;i++)
{   
/*SpMV operations happen here*/
PAPI_read_counters( counters, 2 );

    if ( (retval = PAPI_read_counters(counters, 2)) != PAPI_OK)
    {
    printf("Papi fails!\n");
    }

}

PAPI_stop_counters( counters, 2 );

}

Answer 1

In general you want to know as much about your error as possible when debugging these sorts of problems. In this case, you're not providing enough information with your retval check. See the PAPI documentation on error-handling for more information, but to start, you may want to try:

retval = PAPI_read_counters(counters, 2));
if (retval != PAPI_OK) {
    fprintf(stderr, "PAPI error %d: %s\n", ret, PAPI_strerror(retval));
}

You can also increase the level of debugging information with PAPI_Set_debug, but you need to make sure that you've got a DEBUG-enabled build of PAPI.