cremi_python-master debug

created at 07-29-2021 views: 5

environment

  • Anaconda,
  • pycharm2020.2
  • python3.6

This post mainly involves the conversion of code python2 and python3, and the use of hdf5 files for subsequent learning and use.

download

cremi_python-master: download

We start from example_write.py, generate our own data, then read from example_read.py, and finally example_evaluation.py understand the whole process,

example_write.py debugging

1. Call changes between files

Run1. Call changes between files
Run example_write.py. An error occurred:. An error occurred:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "D:\pycharm\file\python36\cremi_python-master_start\cremi\__init__.py", line 1, in <module>
    from Annotations import *
ModuleNotFoundError: No module named 'Annotations'

According to the error message, the error is found in cremi_init_.py

from Annotations import *
from Volume import *
from io import *
from evaluation import *

The program shows that the call package cannot be found, which is caused by the difference between python2 and python3. Since Annotations, Volume, io, evaluation and other files are in the cremi/ folder, change the program to:

from .Annotations import *
from .Volume import *
from .io import *
from .evaluation import *

In the same way, change in cremi\io_init_.py:

from CremiFile import *

To

from .CremiFile import *

change in cremi\evaluation_init_.py:

from Clefts import *
from NeuronIds import *
from SynapticPartners import *
from border_mask import *

to

from .Clefts import *
from .NeuronIds import *
from .SynapticPartners import *
from .border_mask import *

At the same time, check the call of each python file and change the corresponding path.

2.print usage difference

Run the program again, an error occurs

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "D:\pycharm\file\python36\cremi_python-master_start\cremi\__init__.py", line 3, in <module>
    from .io import *
  File "D:\pycharm\file\python36\cremi_python-master_start\cremi\io\__init__.py", line 1, in <module>
    from .CremiFile import *
  File "D:\pycharm\file\python36\cremi_python-master_start\cremi\io\CremiFile.py", line 40
    print "overwriting existing dataset"
                                       ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("overwriting existing dataset")?
python-BaseException

This is due to the difference in the use of the python version of print, python2 can be without parentheses, python3 needs to be parentheses
Add parentheses to all print methods, for example:

change

print "overwriting existing dataset"

to

print("overwriting existing dataset")

3. problem with data writing

An error occurred after running

Traceback (most recent call last):
  File "D:\Anaconda\program\envs\maskrcnn\lib\site-packages\h5py\_hl\group.py", line 136, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "D:\Anaconda\program\envs\maskrcnn\lib\site-packages\h5py\_hl\dataset.py", line 170, in make_new_dset
    dset_id.write(h5s.ALL, h5s.ALL, data)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5d.pyx", line 222, in h5py.h5d.DatasetID.write
  File "h5py\_proxy.pyx", line 132, in h5py._proxy.dset_rw
  File "h5py\_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite
OSError: Can't write data (no appropriate function for conversion path)

The generated data is difficult to write into the hdf file. Perform debugging and find that the problem occurs in the __create_dataset function of CremiFile.py. The error location is in line 36 of the file:

self.h5file.create_dataset(path, data=data, dtype=dtype, compression=compression)

This statement calls the create_dataset method in the h5py library to write data into the path. In the actual program running, the data cannot be written in, so I check the data type of each data and find:

>>>type(data)
Out[1]: dict_keys

The data in the program is an iterable object of dict_keys. The object cannot be stored directly in the hdf file. Find the source of the data and find that the data comes from the ids method of Annotations.py:

def ids(self):
    """Get the ids of all annotations.
    """
    return self.__types.keys()

Since the return of dict.keys() in python3 is no longer the list type, and index is not supported, so here to convert, change the code to:

def ids(self):
    """Get the ids of all annotations.
    """
    return list(self.__types.keys())

In the same way, in the subsequent types of Annotations.py, the locations method is changed to the following form

def types(self):
    """Get the types of all annotations.
    """
    return list(self.__types.values())

def locations(self):
    """Get the locations of all annotations. Locations are in world units, 
    relative to the offset.
    """
    return list(self.__locations.values())

In the write_annotations method of the CremiFile.py file (lines 107 to 108):

change

self.__create_dataset("/annotations/comments/target_ids", data=annotations.comments.keys(), dtype=np.uint64)
self.__create_dataset("/annotations/comments/comments", data=annotations.comments.values(), dtype=h5py.special_dtype(vlen=unicode))

to

self.__create_dataset("/annotations/comments/target_ids", data=list(annotations.comments.keys()), dtype=np.uint64)
self.__create_dataset("/annotations/comments/comments", data=list(annotations.comments.values()), dtype=h5py.special_dtype(vlen=unicode))

4. problem with the encoding form

Traceback (most recent call last):
  File "D:\pycharm\program\PyCharm 2020.2\plugins\python\helpers\pydev\pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "D:\pycharm\program\PyCharm 2020.2\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/pycharm/file/python36/cremi_python-master_start/example_write.py", line 36, in <module>
    file.write_annotations(annotations)
  File "D:\pycharm\file\python36\cremi_python-master_start\cremi\io\CremiFile.py", line 103, in write_annotations
    self.__create_dataset("/annotations/types", data=annotations.types(), dtype=h5py.special_dtype(vlen=unicode), compression="gzip")
NameError: name 'unicode' is not defined

The place of error is

self.__create_dataset("/annotations/types", data=annotations.types(), dtype=h5py.special_dtype(vlen=unicode), compression="gzip")

This is because the stored data type needs to be determined in the storage process, python3 vlen = str, python 2 vlen=unicode.

Change the parameter to

vlen=str

At this point, the program can run successfully.

example_read.py debugging

1. Encoding error

After running, an error occurred

Traceback (most recent call last):
  File "D:\pycharm\program\PyCharm 2020.2\plugins\python\helpers\pydev\pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "D:\pycharm\program\PyCharm 2020.2\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/pycharm/file/python36/cremi_python-master_start/example_read.py", line 47, in <module>
    print(str(id) + " of type " + type + " at " + str(np.array(location)+np.array(annotations.offset)))
TypeError: must be str, not bytes

This is due to the wrong data type of type:

change 

print(str(id) + " of type " + type + " at " + str(np.array(location)+np.array(annotations.offset)))

to

print(str(id) + " of type " + str(type, encoding='utf-8') + " at " + str(np.array(location)+np.array(annotations.offset)))

At this point, the program can run successfully, the results of the operation:

Has raw: True
Has neuron ids: True
Has clefts: True
Has annotations: True
Read raw: <cremi.Volume.Volume object at 0x000001F7CD0D8BA8>, resolution [40.  4.  4.], offset (0.0, 0.0, 0.0), comment ""
Read neuron_ids: <cremi.Volume.Volume object at 0x000001F7CD0D8D30>, resolution [40.  4.  4.], offset (0.0, 0.0, 0.0), comment "just ones"
Read clefts: <cremi.Volume.Volume object at 0x000001F7CD0F9828>, resolution [40.  4.  4.], offset (0.0, 0.0, 0.0), comment "just zeros"
Read annotations:
0 of type presynaptic_site at [ 8. 14. 45.]
1 of type presynaptic_site at [ 25.  62. 100.]
2 of type presynaptic_site at [39. 12. 72.]
3 of type presynaptic_site at [26. 21. 57.]
4 of type postsynaptic_site at [37. 43. 78.]
5 of type postsynaptic_site at [54.  1. 28.]
6 of type postsynaptic_site at [ 42.  19. 100.]
7 of type postsynaptic_site at [81. 82. 47.]
Pre- and post-synaptic partners:
0 -> 4
1 -> 5
2 -> 6
3 -> 7

Process finished with exit code 0

example_evaluation.py debugging

1. Data type modification

Run, replace the evaluated truth data and test data with the example.hdf we generated, and an error occurs after running:

Traceback (most recent call last):
  File "D:\pycharm\program\PyCharm 2020.2\plugins\python\helpers\pydev\pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "D:\pycharm\program\PyCharm 2020.2\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/pycharm/file/python36/cremi_python-master_start/example_evaluation.py", line 13, in <module>
    adapted_rand = neuron_ids_evaluation.adapted_rand(test.read_neuron_ids())
  File "D:\pycharm\file\python36\cremi_python-master_start\cremi\evaluation\NeuronIds.py", line 63, in adapted_rand
    return adapted_rand(np.array(segmentation.data), self.gt)
  File "D:\pycharm\file\python36\cremi_python-master_start\cremi\evaluation\rand.py", line 51, in adapted_rand
    p_ij = sparse.csr_matrix((ones_data, (segA[:], segB[:])), shape=(n_labels_A, n_labels_B))
  File "C:\Users\...\AppData\Roaming\Python\Python36\site-packages\scipy\sparse\compressed.py", line 54, in __init__
    other = self.__class__(coo_matrix(arg1, shape=shape))
  File "C:\Users\...\AppData\Roaming\Python\Python36\site-packages\scipy\sparse\coo.py", line 154, in __init__
    self._shape = check_shape((M, N))
  File "C:\Users\...\AppData\Roaming\Python\Python36\site-packages\scipy\sparse\sputils.py", line 278, in check_shape
    new_shape = tuple(operator.index(arg) for arg in args)
  File "C:\Users\...\AppData\Roaming\Python\Python36\site-packages\scipy\sparse\sputils.py", line 278, in <genexpr>
    new_shape = tuple(operator.index(arg) for arg in args)
TypeError: 'numpy.float64' object cannot be interpreted as an integer

After debugging, it was found that the error occurred in the adapted_rand method in the rand.py file:

p_ij = sparse.csr_matrix((ones_data, (segA[:], segB[:])), shape=(n_labels_A, n_labels_B))

The inspection found that n_labels_A and n_labels_B are all float type, the data is wrong, the test found:

import numpy as np
dataset=np.array([1],dtype=np.int64)
data_int=dataset[0]
type(data_int)
Out[5]: numpy.int64
type(data_int+1)
Out[6]: numpy.int64
dataset=np.array([1],dtype=np.uint64)
data_uint=dataset[0]
type(data_uint)
Out[9]: numpy.uint64
type(data_uint+1)
Out[10]: numpy.float64

Continue to test and find:

type(np.uint8(1)+np.int8(1))
Out[3]: numpy.int16
type(np.uint16(1)+np.int16(1))
Out[4]: numpy.int32
type(np.uint32(1)+np.int32(1))
Out[5]: numpy.int64
type(np.uint64(1)+np.int64(1))
Out[6]: numpy.float64

According to my simple understanding, adding np.int64 and np.uint64, the result type needs a signed number, and the range of np.int64 cannot include the addend of np.uint64. The theoretical data type is np.int128, but There is no such data type in the numpy library, so the program will automatically convert it to np.float64, even if it becomes the addition of two integers, the data type becomes float
So convert n_labels_A and n_labels_B:

n_labels_A = (np.amax(segA) + 1).astype(np.int64)
n_labels_B = (np.amax(segB) + 1).astype(np.int64)

2. Data reading

Continue to run the program, a warning may appear:

D:\pycharm\file\python36\cremi_python-master_start\cremi\evaluation\Clefts.py:11: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  self.truth_clefts_invalid = truth_clefts.data.value == 0xfffffffffffffffe
D:\pycharm\file\python36\cremi_python-master_start\cremi\evaluation\Clefts.py:13: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  self.test_clefts_mask = np.logical_or(test_clefts.data.value == 0xffffffffffffffff, self.truth_clefts_invalid)
D:\pycharm\file\python36\cremi_python-master_start\cremi\evaluation\Clefts.py:14: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  self.truth_clefts_mask = np.logical_or(truth_clefts.data.value == 0xffffffffffffffff, self.truth_clefts_invalid)

The read method uses dataset[:] as much as possible instead of dataset.value.
Modify the __init__ method in Clefts.py as follows:

self.truth_clefts_invalid = truth_clefts.data[:] == 0xfffffffffffffffe
self.test_clefts_mask = np.logical_or(test_clefts.data[:] == 0xffffffffffffffff, self.truth_clefts_invalid)
self.truth_clefts_mask = np.logical_or(truth_clefts.data[:] == 0xffffffffffffffff, self.truth_clefts_invalid)

The problem is solved, run the program, you can see:

Computing VOI...
Computing RAND...
Neuron IDs
==========
    voi split   : 0.0
    voi merge   : 0.0
    adapted RAND: 0.0
Clefts
======
    false positives: 0
    false negatives: 0
    distance to ground truth: {'mean': 0.0, 'std': 0.0, 'max': 0.0, 'count': 100000, 'median': 0.0}
    distance to proposal    : {'mean': 0.0, 'std': 0.0, 'max': 0.0, 'count': 100000, 'median': 0.0}
Computing matching costs...
16 potential matches found
Finding cost-minimal matches...
4 matches found
Synaptic partners
=================
    fscore: 1.0
created at:07-29-2021
edited at: 07-29-2021: