This paper presents a Bayesian framework to construct non-linear, parsimonious, shallow models for multitask regression. The proposed framework relies on the approximation of an RBF kernel by a linear combination of Random Fourier Features (RFFs). The main idea is to combine both primal and dual views of the same model under a single Bayesian formulation suited for multitask problems. From the dual point of view (kernel methods), the proposed formulation facilitates the introduction of prior domain knowledge through the RBF kernel parameter. From the primal perspective (RFF), the new formulation helps control overfitting and enables a parsimonious overall model (all the tasks share the same set of RFFs selected within a joint Bayesian optimisation). The experimental results show that this framework can lead to significant performance improvements compared to the state-of-the-art methods in multitask nonlinear regression.