HypreParMatrix printed format details

[M8kmyday]

I am saving multiple HypreParMatrix matrices using the Print() method, then reading the results into a c-language program for further processing using complex Petsc and Slepc with Petsc matrices. For a sequential Print(), the format is
row column value
and reading this and placing the values into Petsc Mat's is working fine and giving correct results. I am now working on using more than one rank, and the Print() method is breaking the HypreParMatrix into multiple files. The format is
i j value
but the i and j do not directly indicate the row and column. For example, here is a snippet writing with a single rank:
0 2735 0 2735
0 0 1.08466983310771e-01
0 2356 1.05181854614385e-02
0 2355 -7.42156317001008e-02
0 2354 7.97578195137156e-02
0 2353 -4.11964569354750e-02
0 2352 -2.66776234624309e-02
0 927 -8.04891801767019e-04
0 928 -2.93152666654374e-02
0 929 -2.26686555987002e-02
0 926 -8.26071976800703e-03
0 925 1.09895109549185e-02
0 924 -2.07156019880894e-02
0 1205 2.35306230243572e-02
0 1204 -1.20511309124902e-02
0 1203 2.31517936846435e-03
0 1202 1.59511656150841e-02
0 1201 -1.37900224493088e-02
0 1200 -1.43602114243132e-02
0 6 1.77192030824521e-03
0 7 -2.79368876023576e-03
0 8 -2.37459349809672e-03
0 5 -4.44239253034057e-03
0 4 -7.89813175095653e-04
0 3 3.69250178797575e-03
0 2 -6.48006743608447e-04
0 1 -2.73559079331549e-02
0 2357 1.24736675681853e-02
and here it the same HypreParMatrix written with 2 ranks with the snippet from file *.00000:
0 1391 0 1391
0 0 1.08466983310771e-01
0 1210 1.05181854614385e-02
0 1209 -7.42156317001008e-02
0 1208 7.97578195137156e-02
0 1207 -4.11964569354750e-02
0 1206 -2.66776234624309e-02
0 480 -8.04891801767019e-04
0 481 -2.93152666654374e-02
0 482 -2.26686555987002e-02
0 479 -8.26071976800703e-03
0 478 1.09895109549185e-02
0 477 -2.07156019880894e-02
0 629 2.35306230243572e-02
0 628 -1.20511309124902e-02
0 627 2.31517936846435e-03
0 626 1.59511656150841e-02
0 625 -1.37900224493088e-02
0 624 -1.43602114243132e-02
0 6 1.77192030824521e-03
0 7 -2.79368876023576e-03
0 8 -2.37459349809672e-03
0 5 -4.44239253034057e-03
0 4 -7.89813175095653e-04
0 3 3.69250178797575e-03
0 2 -6.48006743608447e-04
0 1 -2.73559079331549e-02
0 1211 1.24736675681853e-02
So the data is the same, but the i j indices are different. Farther down the files, the i j become quite different between the two files. I cannot figure out how to get the actual row and column numbers from the multi-rank *.xxxxx files. I have reviewed the source code for hypre_ParCSRMatrixReadIJ and hypre_ParCSRMatrixPrintIJ, but so far, I'm not able to reverse-engineer the needed algorithm.

If someone could provide hints or information on any one of the following, I can get going again:

1. The algorithm to calculate the row and column from the i j indices in the multi-rank *.xxxxx files.
2. A pointer to a routine that imports a multi-rank HypreParMatrix matrix into a Petsc Mat. [I have searched, but no luck.]
3. Documentation on the structure and operation of a HypreParMatrix so that I can reverse engineer hypre_ParCSRMatrixReadIJ to get the needed algorithm. [No luck with a search.]
4. Some other idea about how to read a multi-rank HypreParMatrix file set into a Petsc Mat.

[liruipeng]

If you use hypre_ParCSRMatrixPrintIJ, I, J are global row and column indices.