Event-based Vision Sensors (EVSs) exhibit advantages such as high temporal resolution and low data redundancy, rendering them promising for applications in high-speed vision and edge computing scenarios. However, the sparsity and irregularity of event data in both spatial and temporal domains, coupled with the presence of noise, make it difficult for traditional processing methods based on full-pixel sampling and regular memory access to be efficiently adapted to such data, which restricts the real-time performance of EVS-based systems. To address this problem, this paper proposes a software–hardware co-design scheme for event image processing. On the software side, event streams are generated from frame-by-frame luminance differences, and a combination of density clustering, multi-level denoising, and convolution kernel weighting processing is adopted to effectively compress the event scale and enhance structural stability. On the hardware side, a sparse convolution acceleration architecture is designed, where a sparse controller performs block-wise scheduling of multiple processing element units to execute convolution operations only on valid data regions. Experimental results demonstrate that the software-based sparsification processing reduces the number of pixels involved in computation by approximately 95% compared with the unprocessed case; under the same convolution kernel settings, the overall computation achieves a speedup of about 15–20 times relative to hardware acceleration-free schemes. In the FPGA implementation, the system attains a maximum operating frequency of 260 MHz, which verifies the feasibility of the proposed software–hardware co-design architecture for real-time event image processing.