Unable to scroll unless on a control

I currently have a scrollview with controls inside. However on Windows Phone when I try to scroll the page up and down, the scroll doesn't work unless I am touching a control inside that scroll view.

<ScrollView>
        <StackLayout Spacing="10">
          <StackLayout Padding="0,10,0,0">
            <Label Text="Date" Style="{StaticResource LabelStyle}" VerticalOptions="End"></Label>
            <StackLayout Orientation="Horizontal" VerticalOptions="Start">
              <DatePicker Date="{Binding Date}" HorizontalOptions="Fill" />
            </StackLayout>
          </StackLayout>

etc.

If I click on empty space on the screen between controls in my StackLayout, it won't scroll up or down. I have to actually click the Label or any control before the scroll works. Does anyone know why this is happening?