CUDA Tips: nvcc’s -code, -arch, -gencode
Introduction People may feel confused by the options of -code, -arch, -gencode when compiling their CUDA codes. Although the official guidance explains the d...
Recent posts
Expected Data Types in Mixed Precision Cheatsheet
When training neural networks with the Keras API, we care about the data types and computation types since they are relevant to the convergence (numeric stab...
Understanding the GeLU Fusion with TF-Grappler Visualization Tool
Introduction This post focuses on the GELU activation and showcases a debugging tool I created to visualize the TF op graphs. The Gaussian Error Linear Unit,...
Demystifying the BatchNorm-Add-ReLU Fusion
Introduction My previous post, “Demystifying the Conv-Bias-ReLU Fusion”, has introduced a common fusion pattern in deep learning models. This post, on the ot...
Sparse Data Structure: Sorting Indices with Any Sorter + Custom Comparators
Introduction Recently, I am working on a project regarding sparse tensors in Tensorflow. Sparse tensors are used to represent tensors with many zeros. To sav...