Image sensors with integrated signal processing – so called “Vision Chips” – allow for execution of computationally intensive processing steps directly after image acquisition. Programmable systems, whose functional units may be utilized in a flexible manner for various image processing tasks, require a flexible, modular toolchain. A programming environment, consisting of an assembler supporting ASIP-based (Application Specific Instruction Set Processor) control units and a Python translator supporting a subset of the Python programming language, will be presented. Library elements are used to further abstract the behavior of the underlying Vision-System-on-Chip (VSoC). For a concrete task, both VSoC-internal and conventional processing steps can be implemented within the same project. When combined with established libraries such as OpenCV, VSoC-internal processing close to the sensor becomes a powerful tool for holistic vision task design.